apache / apache/airflow

Prevent stale task triggers from blocking healthy workloads

Open
#69,841 1 comment 0 reactions 0 assignees View on GitHub
area:Triggerer kind:bug priority:high
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Apache Airflow version

3.3.0 and current `main`

### What happens

The triggerer supervisor loads the pinned serialized Dag for every task-associated trigger so it can determine whether the task uses `start_from_trigger`. If that serialized Dag exists but no longer contains the deferred task, `get_task()` raises `TaskNotFound` from workload construction. The exception escapes the per-batch loop, so one stale task instance prevents unrelated healthy triggers in the same batch from starting and can repeatedly restart the triggerer supervisor.

### Sanitized reproducer

1. Create two deferred task instances with ordinary triggers.
2. Leave one task instance linked to a Dag version whose serialized Dag no longer contains that task ID.
3. Leave the second task instance and its pinned Dag version valid.
4. Let the triggerer claim both trigger rows in one polling cycle.

The first call to `LazyDeserializedDAG.get_task()` raises `TaskNotFound`. The healthy trigger is not constructed during that supervisor pass.

### Expected behavior

A stale task instance must not abort construction of unrelated trigger workloads. A backportable fix can treat this narrow lookup failure as an ordinary context-free deferral, matching behavior before triggerer-side `start_from_trigger` context loading was introduced.

The durable fix should persist whether each trigger requires Dag context. Ordinary deferrals can then avoid serialized-Dag lookup entirely, direct-to-triggerer tasks can require their pinned Dag data, and legacy rows can self-classify once. If a context-required row points to a serialized Dag that lacks its task, the task instance should become `REMOVED` and unlink from the trigger without blocking healthy workloads.

### Acceptance criteria

- One stale task-associated trigger cannot abort the triggerer supervisor batch.
- Ordinary deferrals do not query or deserialize their Dag.
- Context-required triggers never run with missing or unrendered Dag context.
- Legacy rows remain safe during rolling upgrades.

---
Drafted-by: Codex (GPT-5) (no human review before posting)

Contributor guide

Open the contributing guide

Research direction

Start at the triggerer supervisor's per-batch workload construction and trace LazyDeserializedDAG.get_task() through TaskNotFound handling. Verify behavior with stale and healthy trigger rows, including legacy rows and context-required triggers. Done means one stale row cannot abort the batch, ordinary deferrals avoid Dag deserialization, and missing context-required tasks are removed safely.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.