Different `max_active_runs` for scheduled vs manual runs
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
Hi
We have a use case where a DAG is scheduled similar to `ContinuousTimetable`, where it should run as often as possible, but it shouldn't be scheduled to run more than one at a time.
The same DAG can be started manually, with a set of parameters to override the default behaviour. When someone decides to start manually, it is OK to have 2 or more of them running concurrently.
I can't find any way to implement this, setting `max_active_runs=1`, constrains also the manual runs.
I would like a new parameter (`max_active_scheduled_runs`?), or some new extension point in the `TimeTable` API, that can take all ongoing runs into consideration and use that to withhold scheduling new runs.
### Use case/motivation
Some example use cases
* The scheduled runs are running a complete test suite that takes many hours. Using Params, a user can select one specific test to run, that will only take minutes to run.
* Shortening lead times. Imagine a run taking 1 hour. Half an hour in you realize you need to do a new run with some new data that became available. Instead of waiting for the current run to finish or aborting the current run, you can manually override to start a second run in parallel. Always having a half-hour cadence active is not desirable due to cost.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reading how max_active_runs constrains DAG runs and how the ContinuousTimetable and Timetable API participate in scheduling. Compare scheduled and manually triggered runs, then define an extension or parameter that limits only scheduled concurrency while preserving manual overrides. Done means the behavior is specified and covered for both run types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100