Allow different parallelism configuration value for multi-executor environment
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
We should be able to configure parallelism limit separately for each executor.
Or at least for local / k8s pair
### Use case/motivation
With the practical `LocalExecutor` <-> `KubernetesExecutor` pair for example we can have tons of tasks running freely in the k8s environment, but we need to limit the local executor to some small value to not overload the scheduler pod. And it's very tedious to run an extra scheduler for k8s environment unleashing.
I believe, that for other multiple configurations idea of total parallelism limitation would depend on used executor too.
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
No files or tests are named. Start by reviewing how the LocalExecutor and KubernetesExecutor share the parallelism limit, then define separate limits for the local and Kubernetes environments and verify that the scheduler pod is no longer constrained by Kubernetes workload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100