Operator extra links don't work per attempt
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
**TL;DR — operator extra links don't work per attempt.** After a retry, asking for an earlier
attempt's link returns the latest attempt's URL, so the button under attempt 1 opens attempt
2's logs.
It hits anyone whose link carries a job id the service mints per submission — every bundled
provider linking to per-attempt logs, so EMR, Glue, Databricks, Dataproc, Livy.
### Why
A link is one XCom row under `BaseOperatorLink.xcom_key` (`_link_` by default).
1. `XComOperatorLink.get_link` gets a `TaskInstanceKey` and ignores its `try_number`.
2. That row only ever holds the latest attempt: the run endpoint clears every XCom for the
task before each attempt ([task_instances.py#L285-L293](https://github.com/apache/airflow/blob/537feafb3697338b3988d826b9c9b45da06b1f23/airflow-core/src/airflow/api_fastapi/execution_api/routes/task_instances.py#L285-L293)),
and `XComModel.set` deletes before it inserts.
\#65661 made the endpoint resolve the requested attempt, which is what makes the wrong answer
reachable from the UI.
No workaround exists: `xcom_key` is a plain property with no access to `ti_key`, the `xcom`
table has no `try_number` column, and the clear takes every key regardless.
### Reproduce
Run a task with an extra link, make it retry, ask the extra-links endpoint for
`try_number=1`. You get attempt 2's URL, and the metastore holds one `_link_*` row.
### Fix
#71522 — each attempt's link goes to the task state store, which a retry does not clear. That
keeps the XCom clear absolute instead of carving an exemption into it, and it is where
cross-retry state already lives (#69914, #71211).
Two alternatives were drafted and closed: a row per attempt in XCom (#71518) and the same with
the link opting in (#71521). #71518 is the one to revive if this needs to reach the 3.2 line,
since the state store landed after 3.2.2.
### Apache Airflow version
3.2.2. The code path is unchanged on `main` at 537feafb3697338b3988d826b9c9b45da06b1f23.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start at XComOperatorLink.get_link and task_instances.py#L285-L293, then inspect BaseOperatorLink.xcom_key and XComModel.set to understand why try_number is lost. Compare the proposed task-state-store approach in #71522; done means the extra-links endpoint returns the URL for the requested attempt while retry XCom clearing remains intact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100