apache / apache/airflow

Support different plugins_folder setting in helm chart when airflow version > 2.0.0

Open
#39,562 2 comments 0 reactions 0 assignees View on GitHub
area:helm-chart area:plugins kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

When configuring Airflow using a Helm chart, the `core` section allows you to set the `plugins_folder`. This setting is applied equally across all web servers and schedulers. Many users utilize git-sync, and it's common for the plugins' path to be located within a git-synced repository.

With Airflow version 2.0.0 and above, Dag Serialization is enabled by default, which means that the web server cannot use a git-sync sidecar. This is generally not an issue when developing custom operators, but it can cause confusion if you want to customize the web UI and the plugins folder on the web server is not synced through git-sync.

Therefore, it would be beneficial to guide the explicit setting of the `plugins_folder` in the Helm chart, allowing separate configurations for the web server and scheduler. This approach would enable more flexible management of each component's settings, allowing finer adjustments to meet specific requirements.

### Use case/motivation

When attempting to customize the Airflow Web UI based on AirflowPlugin, I followed the guidance to write scripts in the plugins directory. However, with Airflow version 2.0.0 and above, there was confusion in using the webserver plugins because git-sync does not operate as a sidecar for the webserver.

As documented at [Airflow's Helm Chart Guide](https://airflow.apache.org/docs/helm-chart/stable/manage-dags-files.html#notes-for-combining-git-sync-and-persistence), it is generally not recommended to use git-sync with persistence together. This recommendation has significantly limited the possible workarounds to address this issue.

### Related issues

[Airflow Slack](https://apache-airflow.slack.com/archives/CCQ7EGB1P/p1715406951023049)

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the Helm chart's core settings and the webserver and scheduler configuration paths, then review the linked guidance on git-sync and persistence. Done means the chart supports and documents separate plugins_folder values for the webserver and scheduler when using Airflow 2.0.0 or later.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, python
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.