Define safe retry semantics for Execution API Dag run trigger and reset operations
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Body
- Follow-up to https://github.com/apache/airflow/pull/66911.
`DagRunOperations.trigger()` sends `POST /execution/dag-runs/{dag_id}/{run_id}` through the generic Execution API retry layer, but the operation is not idempotent:
1. The server creates the child Dag run.
2. The client loses the successful response and retries.
3. The retry receives 409 Conflict.
4. The parent task fails and does not write the trigger_run_id XCom, although the child run was created.
The client-side recovery approach explored in #66911 leaves ambiguity: finding a run after a transport failure cannot distinguish successful creation by this request from a pre-existing run whose conflict response was lost. Returning success in both cases changes conflict and skip_when_already_exists behavior. A pre-check or bounded POST/count loop does not eliminate this race.
We are deferring that approach to address the server contract in a follow-up.
Proposed scope
- Define how the server distinguishes a retry of the same trigger/reset operation from a separate operation targeting the same run.
- Move reset_dag_run handling into the Execution API trigger route, performing the equivalent of clearing the existing run server-side.
- Provide replay protection so retrying a completed operation returns its original outcome without creating or clearing again.
- Preserve existing conflict, skip, and reset semantics for separate operations.
- Preserve bounded retries for failures known to occur before the request is sent, and define recovery for ambiguous transport failures.
- Keep the successful trigger path to one request.
Moving reset handling server-side alone does not make retries safe: a lost response followed by a retry could clear the run again after tasks have started.
A stable operation ID with durable result tracking is one possible design. Whether this requires a dedicated table remains open. The design should specify atomicity, concurrent requests, operation-ID lifetime, retention, and compatibility with older clients.
Acceptance criteria
- A lost successful trigger response can be recovered without incorrectly failing the parent or omitting trigger_run_id.
- A lost conflict response cannot turn a separate operation into success or bypass configured fail/skip behavior.
- Retrying a reset does not clear the run twice, including when another intentional reset occurs before an older retry arrives.
- Concurrent requests and delayed commits preserve the agreed replay and conflict semantics.
- Tests cover transport failures, replay, concurrent creation/reset, parent task state, XCom behavior, and client/server compatibility.
### Committer
- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Contributor guide
Assessment
This issue has not been assessed yet.