apache / apache/airflow

When triggering a paused DAG, ensure the manually triggered run starts before a scheduled run

Open
#55,675 11 comments 0 reactions 0 assignees View on GitHub
area:core area:scheduler kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

For the description below, assume that the DAG:
* is scheduled to run once a day
* has been paused more than one day
* has `max_active_runs=1`
* was not running when it was paused (e.g. all tasks are success or failed)

When triggering a paused DAG, it would be nice if the manually triggered run starts before any scheduled run.

If the DAG has parameters, there is a "Unpause DAG when triggered" toggle, but this doesn't help.

This is the behavior currently seen, running with the official Helm chart, using the Celery scheduler:
* With "Unpause DAG when triggered" = true: regularly scheduled run starts running, the manually triggered run can run only after the other one completes (naturally, or because marked success / failure)
* With "Unpause DAG when triggered" = false: only the manually triggered run is scheduled - until the DAG is unpaused, at which time the regularly scheduled run starts running, and the manually triggered run can run only after the other one completes (naturally, or because marked success / failure)

### Use case/motivation

We have an Integration environment, but most of the DAGs are usually paused on Integration. Some of the DAGs process a lot of data, and take a long time to run. These DAGs are run on Integration only occasionally, for example when testing changes. These DAGs often have parameters to reduce the amount of data processed (e.g. only process the list of ids (1,2,3), or only process the source data within a given date/time range).

It's an annoyance for us, and counterintuitive, that when we trigger a paused DAG, the manually triggered run, with the parameters we have specified, is not the one that starts running - it's the regularly scheduled DAG run that goes first (obviously, without the parameters that have been specified). We have to then mark the regularly scheduled run as fail / success, before our parameterized manual run starts.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Reproduce the case with the official Helm chart and Celery scheduler: use a paused daily DAG with max_active_runs=1, an overdue scheduled run, and a parameterized manual trigger. Compare both settings of the "Unpause DAG when triggered" toggle; done means the manually triggered run starts before the scheduled run.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.