apache / apache/airflow

Pausing a DAG stalls already-running DagRuns instead of only blocking new ones

Open
#71,381 1 comment 0 reactions 0 assignees View on GitHub
area:scheduler kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

When a DAG is paused while it has an active (running) DagRun — e.g. one started via manual trigger before the pause, or a manual trigger on an already-paused DAG that got an initial burst of tasks dispatched — the scheduler stops scheduling further task instances for that run. The run doesn't fail and logs no error; it simply stalls indefinitely in `running` state with remaining tasks stuck in `scheduled`, `queued_when=None`, never sent to the executor.

This happens because `is_paused` is checked as a single gate that governs both "should new DagRuns be created" and "should ready task instances in existing DagRuns be scheduled." The two are conflated.

### Use case/motivation

Pause is a reasonable way to say "don't start new runs of this DAG." It's not intuitive that it also freezes runs already in flight — especially since:
- No error, warning, or DAG/task state reflects why the run stopped progressing.
- A manually triggered run against a paused DAG appears to start successfully (initial tasks execute), masking the problem until the run has been silently stuck for several minutes.
- The eventual fix (unpause) has a side effect the operator may not want: it also releases scheduling for *new* runs of that DAG, when the operator's only goal was to unstick the one in-flight run.

### Suggested change

Preserve pre-run pause semantics: `is_paused` should gate DagRun *creation*, not task-instance scheduling for DagRuns already in `running` state. A DagRun that starts (however it started) while unpaused, or that gets tasks dispatched before a pause takes effect, should be allowed to run to completion. If the DAG is still paused when the run finishes, subsequent runs are blocked as expected — pause state before and after the run is preserved; only the in-flight run is exempted.

At minimum, surfacing this in the UI/logs (e.g., a task/run banner: "this DagRun's DAG is paused; scheduling is suspended") would prevent the silent-stall confusion, even if the gating behavior itself is kept as-is by design.

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Trace the scheduler paths that use is_paused, separating DagRun creation from task-instance scheduling for DagRuns already in running state. Reproduce a paused-DAG run with remaining tasks in scheduled state, then verify that an in-flight run can complete while new runs remain blocked when the DAG is paused.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.