apache / apache/airflow

Proposal: Per-Bundle DAG Processor Deployments in Helm Chart

Open
#61,037 3 comments 0 reactions 0 assignees View on GitHub
area:dag-bundles area:helm-chart kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

## Summary

Currently, the Helm chart processes all DAG bundles in a single DAG Processor deployment. This proposal adds an option to create separate deployments for each DAG bundle defined in `dagBundleConfigList`, enabling independent resource isolation and scaling per bundle.
https://github.com/apache/airflow/blob/0cf89fa362e72898b718fe6ff2fc8caa0b4ba45f/chart/values.yaml#L2274-L2296

## Motivation

### Problems

1. **Resource Contention**: When multiple DAG bundles run in a single DAG Processor deployment, if one bundle consumes excessive CPU/memory, it impacts the parsing performance of other bundles.

2. **Parsing Time Delays**: When a specific bundle has many or complex DAG files, parsing that bundle takes longer and can delay parsing of other bundles.

3. **Scaling Limitations**: Even when bundles have different resource requirements or priorities, all bundles must currently be handled by the same deployment.

4. **Lack of Failure Isolation**: Issues occurring in one bundle can affect the entire DAG Processor.

### Use Cases
- Loading DAGs from multiple Git repositories (different bundle per repository)
- When specific bundles require higher priority or more resources

## Proposed Solution

### Feature Overview

Add a new `deployPerBundle` section to the `dagProcessor` configuration to create separate Kubernetes deployments for each bundle. When `deployPerBundle.enabled` is set to `true`, a separate deployment will be created for each bundle in `dagBundleConfigList`.

The `airflow dag-processor --bundle-name` (or `-B`) option is already an existing feature in Airflow core. This proposal only requires implementing the Helm chart configuration to utilize this existing functionality, making it straightforward to deploy per-bundle DAG processors.

### values.yaml Changes

```yaml
dagProcessor:
enabled: ~

dagBundleConfigList:
- name: dags-folder
classpath: "airflow.dag_processing.bundles.local.LocalDagBundle"
kwargs: {}
# ... more bundles

# Per-bundle deployment option
# When enabled, creates a separate deployment for each bundle in dagBundleConfigList
deployPerBundle:
enabled: false
# Command args template for per-bundle deployments
# {{ bundleName }} will be replaced with the actual bundle name
args: ["bash", "-c", "exec airflow dag-processor --bundle-name {{ bundleName }}"]

# Per-bundle specific overrides (optional)
bundleOverrides:
dags-folder:
replicas: 2
resources:
requests:
memory: "2Gi"
cpu: "1000m"
```

### Backward Compatibility

- When `deployPerBundle.enabled: false` (default), existing behavior is maintained
- Per-bundle deployments are only created when `deployPerBundle.enabled: true`
- Existing `dagProcessor` settings (like `replicas`, `resources`, etc.) are used as defaults for per-bundle deployments
- The `deployPerBundle.args` replaces the default `args` when `deployPerBundle.enabled` is `true`
- Bundle-specific configuration overrides are possible via `deployPerBundle.bundleOverrides`

## Benefits

1. **Resource Isolation**: Each bundle runs in independent pods, preventing resource contention
2. **Independent Scaling**: Different replica counts can be set per bundle
3. **Failure Isolation**: Issues in one bundle do not affect others
4. **Flexible Resource Allocation**: Different resource requests/limits can be configured per bundle
5. **Easier Monitoring**: Metrics and logs can be separated and tracked per bundle

### Use case/motivation

_No response_

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with chart/values.yaml at the dagBundleConfigList section and review the existing Helm chart configuration for the DAG Processor. Use the existing `airflow dag-processor --bundle-name` option as the entry point; done means enabling the proposed setting creates one deployment per bundle with defaults, overrides, and preserved behavior when disabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, python
Domain
devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.