apache / apache/airflow

Expose reconnect-decision separately from polling, for deferrable operators

Open
#71,485 0 comments 0 reactions 1 assignee Claimed by @amoghrajesh View on GitHub
area:providers kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

`ResumableJobMixin`'s crash-recovery decision logic (reconnect / already-succeeded / terminal-resubmit) lives entirely inside `execute_resumable`, inlined together with the synchronous polling that follows it. There's no way for a caller to get just the decision without also getting blocked on `poll_until_complete`.

This matters for any operator that supports both `deferrable=True` and durable execution, `GlueJobOperator` included. On the deferrable path, the operator calls `submit_job` directly from `execute()`, bypassing the mixin's `task_state_store` read entirely — nothing is ever persisted there for a deferred run. So a deferrable retry has no cheap way to check "is there already a run I can reconnect to" and instead always falls through to the operator's own bootstrap fallback (for Glue, a full paginated `get_job_runs` scan), even though the run id could have been persisted to `task_state_store` before `defer()` and read back cheaply on retry.

Fixing this requires the mixin to expose its reconnect-decision step (read stored id, call `get_job_status`, apply `is_job_active`/`is_job_succeeded`) as something callable on its own, separate from the polling loop in `execute_resumable`. Once that exists, a deferrable operator could:

1. Call the decision step before `defer()`.
2. If nothing to reconnect to, submit fresh and persist the id to `task_state_store`.
3. On a deferrable retry, call the decision step again before deciding whether to resubmit.

This would let deferrable retries reconnect via the cheap store lookup instead of an operator-specific bootstrap scan, for every current and future `ResumableJobMixin` port that supports `deferrable=True`.

Raised during review of #71211 (kaxil, https://github.com/apache/airflow/pull/71211#discussion_r3764425769 — see comment thread for full context).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.