Proposal: Per-Bundle DAG Processor Deployments in Helm Chart
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Description
## Summary
Currently, the Helm chart processes all DAG bundles in a single DAG Processor deployment. This proposal adds an option to create separate deployments for each DAG bundle defined in `dagBundleConfigList`, enabling independent resource isolation and scaling per bundle.
https://github.com/apache/airflow/blob/0cf89fa362e72898b718fe6ff2fc8caa0b4ba45f/chart/values.yaml#L2274-L2296
## Motivation
### Problems
1. **Resource Contention**: When multiple DAG bundles run in a single DAG Processor deployment, if one bundle consumes excessive CPU/memory, it impacts the parsing performance of other bundles.
2. **Parsing Time Delays**: When a specific bundle has many or complex DAG files, parsing that bundle takes longer and can delay parsing of other bundles.
3. **Scaling Limitations**: Even when bundles have different resource requirements or priorities, all bundles must currently be handled by the same deployment.
4. **Lack of Failure Isolation**: Issues occurring in one bundle can affect the entire DAG Processor.
### Use Cases
- Loading DAGs from multiple Git repositories (different bundle per repository)
- When specific bundles require higher priority or more resources
## Proposed Solution
### Feature Overview
Add a new `deployPerBundle` section to the `dagProcessor` configuration to create separate Kubernetes deployments for each bundle. When `deployPerBundle.enabled` is set to `true`, a separate deployment will be created for each bundle in `dagBundleConfigList`.
The `airflow dag-processor --bundle-name` (or `-B`) option is already an existing feature in Airflow core. This proposal only requires implementing the Helm chart configuration to utilize this existing functionality, making it straightforward to deploy per-bundle DAG processors.
### values.yaml Changes
```yaml
dagProcessor:
enabled: ~
dagBundleConfigList:
- name: dags-folder
classpath: "airflow.dag_processing.bundles.local.LocalDagBundle"
kwargs: {}
# ... more bundles
# Per-bundle deployment option
# When enabled, creates a separate deployment for each bundle in dagBundleConfigList
deployPerBundle:
enabled: false
# Command args template for per-bundle deployments
# {{ bundleName }} will be replaced with the actual bundle name
args: ["bash", "-c", "exec airflow dag-processor --bundle-name {{ bundleName }}"]
# Per-bundle specific overrides (optional)
bundleOverrides:
dags-folder:
replicas: 2
resources:
requests:
memory: "2Gi"
cpu: "1000m"
```
### Backward Compatibility
- When `deployPerBundle.enabled: false` (default), existing behavior is maintained
- Per-bundle deployments are only created when `deployPerBundle.enabled: true`
- Existing `dagProcessor` settings (like `replicas`, `resources`, etc.) are used as defaults for per-bundle deployments
- The `deployPerBundle.args` replaces the default `args` when `deployPerBundle.enabled` is `true`
- Bundle-specific configuration overrides are possible via `deployPerBundle.bundleOverrides`
## Benefits
1. **Resource Isolation**: Each bundle runs in independent pods, preventing resource contention
2. **Independent Scaling**: Different replica counts can be set per bundle
3. **Failure Isolation**: Issues in one bundle do not affect others
4. **Flexible Resource Allocation**: Different resource requests/limits can be configured per bundle
5. **Easier Monitoring**: Metrics and logs can be separated and tracked per bundle
### Use case/motivation
_No response_
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with chart/values.yaml at the dagBundleConfigList section and review the existing Helm chart configuration for the DAG Processor. Use the existing `airflow dag-processor --bundle-name` option as the entry point; done means enabling the proposed setting creates one deployment per bundle with defaults, overrides, and preserved behavior when disabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- helm, kubernetes, python
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100