apache / apache/airflow

Enable Airflow DAG processor horizontally scalable

Open
#32,966 8 comments 1 reaction 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Description

Right now, the airflow DAG processor running in standalone or non standalone mode. Running multiple instances of the DAG processor does duplicate work. It's good idea to run the DAG processor in distributed mode to share the parsing workload.

### Use case/motivation

Running the Airflow at a large scale requires running all the components of the airflow in distributed mode. Right now, the webserver and scheduler can able to scale horizontally. Right now, the DAG processor is vertically scalable but not horizontally scalable. This feature request is to make the DAG processor horizontally scalable.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points. First map the DAG processor's standalone and non-standalone modes and determine how parsing work is currently assigned; done means multiple processor instances can share parsing work without duplicating it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.