apache / apache/airflow

Bound and harden the task-UUID scan fallback

Open
#71,489 7 comments 0 reactions 0 assignees View on GitHub
area:providers good first issue provider:amazon
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

`GlueJobOperator._find_job_run_id_by_task_uuid` (the task-UUID scan fallback, used when neither `task_state_store` nor a cached XCom id has a prior run) has a few pre-existing quality issues, carried over verbatim from `main`, that are worth cleaning up:

1. **Unbounded pagination.** It's a `while True` over `get_job_runs(MaxResults=50)` with no page cap and no age cutoff. The no-match case — the prior attempt died before it ever called `StartJobRun`, which is the common retry shape — is exactly the one that walks the job's entire run history before giving up.
2. **Failure is silently swallowed.** The two `except Exception` blocks around this scan (and the XCom lookup before it) log at `warning` level and return `None`, after which the operator submits fresh — straight into `ConcurrentRunsExceededException` against a run that's still alive. There's no error-level signal telling anyone why.
3. **The `except Exception` is broader than it needs to be.** It should be narrowed to `except ClientError`, so a real bug in the surrounding code doesn't get silently absorbed the same way as a transient AWS-side issue.
4. **Missing IAM documentation.** Anyone relying on this fallback (via `resume_glue_job_on_retry=True` today, or explicit `durable=True` on Airflow <3.3 going forward) needs `glue:GetJobRuns` in their task policy, on top of `StartJobRun`/`GetJobRun`. The docs don't currently call that out.

None of this is new — it's present on `main` today, gated behind the opt-in `resume_glue_job_on_retry` flag. Raised during review of #71211, initially flagged as more urgent because that PR was going to make the scan reachable by default; the default-flip was scoped back to explicit opt-in for Airflow <3.3 (and to a narrow crash-recovery window on 3.3+) in the same PR, so the original urgency no longer applies — but the underlying issues are still real for anyone who does hit this path.

Fix direction: bound the walk (a page cap, or stop once a run's `StartedOn` predates the DAG run), narrow the exception handling, log the fallthrough at `error` level with the actual reason, and document the IAM requirement.

See https://github.com/apache/airflow/pull/71211#discussion_r3764425788 for the original discussion.

Contributor guide

Open the contributing guide

Research direction

Start with GlueJobOperator._find_job_run_id_by_task_uuid and trace the task-UUID scan and preceding XCom lookup. Bound the scan, narrow AWS exception handling, report fallthrough errors with their cause, and update the task-policy documentation to require glue:GetJobRuns; verify the affected operator behavior and documentation changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.