apache / apache/airflow

Define safe retry semantics for Execution API Dag run trigger and reset operations

Open
#72,876 1 comment 0 reactions 1 assignee Claimed by @jason810496 View on GitHub
area:API area:task-sdk type:improvement
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Body

- Follow-up to https://github.com/apache/airflow/pull/66911.

`DagRunOperations.trigger()` sends `POST /execution/dag-runs/{dag_id}/{run_id}` through the generic Execution API retry layer, but the operation is not idempotent:

1. The server creates the child Dag run.
2. The client loses the successful response and retries.
3. The retry receives 409 Conflict.
4. The parent task fails and does not write the trigger_run_id XCom, although the child run was created.

The client-side recovery approach explored in #66911 leaves ambiguity: finding a run after a transport failure cannot distinguish successful creation by this request from a pre-existing run whose conflict response was lost. Returning success in both cases changes conflict and skip_when_already_exists behavior. A pre-check or bounded POST/count loop does not eliminate this race.

We are deferring that approach to address the server contract in a follow-up.

Proposed scope

- Define how the server distinguishes a retry of the same trigger/reset operation from a separate operation targeting the same run.
- Move reset_dag_run handling into the Execution API trigger route, performing the equivalent of clearing the existing run server-side.
- Provide replay protection so retrying a completed operation returns its original outcome without creating or clearing again.
- Preserve existing conflict, skip, and reset semantics for separate operations.
- Preserve bounded retries for failures known to occur before the request is sent, and define recovery for ambiguous transport failures.
- Keep the successful trigger path to one request.

Moving reset handling server-side alone does not make retries safe: a lost response followed by a retry could clear the run again after tasks have started.

A stable operation ID with durable result tracking is one possible design. Whether this requires a dedicated table remains open. The design should specify atomicity, concurrent requests, operation-ID lifetime, retention, and compatibility with older clients.

Acceptance criteria

- A lost successful trigger response can be recovered without incorrectly failing the parent or omitting trigger_run_id.
- A lost conflict response cannot turn a separate operation into success or bypass configured fail/skip behavior.
- Retrying a reset does not clear the run twice, including when another intentional reset occurs before an older retry arrives.
- Concurrent requests and delayed commits preserve the agreed replay and conflict semantics.
- Tests cover transport failures, replay, concurrent creation/reset, parent task state, XCom behavior, and client/server compatibility.

### Committer

- [x] I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.