Enable Airflow DAG processor horizontally scalable
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Description
Right now, the airflow DAG processor running in standalone or non standalone mode. Running multiple instances of the DAG processor does duplicate work. It's good idea to run the DAG processor in distributed mode to share the parsing workload.
### Use case/motivation
Running the Airflow at a large scale requires running all the components of the airflow in distributed mode. Right now, the webserver and scheduler can able to scale horizontally. Right now, the DAG processor is vertically scalable but not horizontally scalable. This feature request is to make the DAG processor horizontally scalable.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
The issue names no files, tests, or entry points. First map the DAG processor's standalone and non-standalone modes and determine how parsing work is currently assigned; done means multiple processor instances can share parsing work without duplicating it.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100