apache / apache/airflow

Different `max_active_runs` for scheduled vs manual runs

Open
#44,336 0 comments 0 reactions 0 assignees View on GitHub
area:scheduler kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Hi
We have a use case where a DAG is scheduled similar to `ContinuousTimetable`, where it should run as often as possible, but it shouldn't be scheduled to run more than one at a time.
The same DAG can be started manually, with a set of parameters to override the default behaviour. When someone decides to start manually, it is OK to have 2 or more of them running concurrently.

I can't find any way to implement this, setting `max_active_runs=1`, constrains also the manual runs.

I would like a new parameter (`max_active_scheduled_runs`?), or some new extension point in the `TimeTable` API, that can take all ongoing runs into consideration and use that to withhold scheduling new runs.

### Use case/motivation

Some example use cases

* The scheduled runs are running a complete test suite that takes many hours. Using Params, a user can select one specific test to run, that will only take minutes to run.
* Shortening lead times. Imagine a run taking 1 hour. Half an hour in you realize you need to do a new run with some new data that became available. Instead of waiting for the current run to finish or aborting the current run, you can manually override to start a second run in parallel. Always having a half-hour cadence active is not desirable due to cost.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading how max_active_runs constrains DAG runs and how the ContinuousTimetable and Timetable API participate in scheduling. Compare scheduled and manually triggered runs, then define an extension or parameter that limits only scheduled concurrency while preserving manual overrides. Done means the behavior is specified and covered for both run types.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.