apache / apache/airflow

Allow different parallelism configuration value for multi-executor environment

Open
#46,787 10 comments 0 reactions 0 assignees View on GitHub
area:Executors-core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

We should be able to configure parallelism limit separately for each executor.

Or at least for local / k8s pair

### Use case/motivation

With the practical `LocalExecutor` <-> `KubernetesExecutor` pair for example we can have tons of tasks running freely in the k8s environment, but we need to limit the local executor to some small value to not overload the scheduler pod. And it's very tedious to run an extra scheduler for k8s environment unleashing.

I believe, that for other multiple configurations idea of total parallelism limitation would depend on used executor too.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reviewing how the LocalExecutor and KubernetesExecutor share the parallelism limit, then define separate limits for the local and Kubernetes environments and verify that the scheduler pod is no longer constrained by Kubernetes workload.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.