apache / apache/airflow

[Helm Chart] Support multiple DagProcessors

Open
#26,494 12 comments 2 reactions 0 assignees View on GitHub
area:helm-chart good first issue kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Airflow 2.4.0 has introduced support for [Support multiple DagProcessors parsing files from different locations](https://github.com/apache/airflow/pull/25935).

However, Helm chart only supports [Provision Standalone Dag Processor](https://github.com/apache/airflow/pull/23711) (or at least will support in Airflow Helm Chart 1.7.0).

### Use case/motivation

For Airflow Helm Chart users take advantage of these new [AIP-43](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-43+DAG+Processor+separation) features, it is necessary to add support for provision multiple DagProcessors in Helm chart.

### Related issues

[Helmchart: Provision Standalone Dag Processor](https://github.com/apache/airflow/pull/23711)
[Support multiple DagProcessors parsing files from different locations](https://github.com/apache/airflow/pull/25935)

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the Helm chart's existing standalone Dag Processor configuration and read the linked Airflow 2.4.0 multiple-DagProcessor change. Done means the chart can provision multiple DagProcessors so users can parse files from different locations.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.