[CT-1475] [Bug] Automatic Backport fails

### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

We currently had to do some backports to `1.1.latest`, `1.2.latest` and `1.3.latest` ([original PR](#6204))

The `1.3.latest` backport worked flawlessly.

However the `1.1.latest` and the `1.2.latest` both failed. When I manually cherry picked the commits I was able to do it without conflict so it's unclear why the cherry picks failed.

Additionally, I was told that if the labels are added _before_ the PR has been merged, the backport will not happen. the labels must be added after the merge is complete. Unsure if this is a bug or just how it works but wanted to note it here as well.

### Expected Behavior

Backports just work - always!

### Steps To Reproduce

See #6204