apache / apache/airflow

airflow.cfg can't pull secrets from LocalFilesystemBackend

Open
#41,512 4 comments 0 reactions 0 assignees View on GitHub
area:core area:secrets kind:documentation
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Apache Airflow version

2.9.3

### If "Other Airflow 2 version" selected, which one?

_No response_

### What happened?

When starting an Airflow instance or checking a property via `airflow config list/get-value` properties that have the `_secret` suffix return their literal value and the main property isn't set at all. The default values instead are applied.

### What you think should happen instead?

According to the docs, the following snippet should retrieve the key `sql_alchemy_conn` from the secret backend (the local filesystem one, in my case) and assign it to the `sql_alchemy_conn` under the `[database]` section.
```
[database]
sql_alchemy_conn_secret = sql_alchemy_conn
```
https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html

### How to reproduce

Reproducing requires only configuring the following properties in airflow.cfg and the .env file in the same directory. The secret backend still works from within a DAG, however, it just seems to not be invoked while parsing the airflow.cfg. I tried bringing the [secrets] section to the top of the airflow.cfg, since by default it comes after [database], but that had no effect.
```
[secrets]
backend = airflow.secrets.local_filesystem.LocalFilesystemBackend
backend_kwargs = {"variables_file_path": "./.env"}

[database]
sql_alchemy_conn_secret = sql_alchemy_conn
```
and an .env file (or yaml or json) like this
```
sql_alchemy_conn=postgresql://airflow_user@localhost/airflow_db
```

### Operating System

Arch linux 6.10.4-arch2-1

### Versions of Apache Airflow Providers

apache-airflow-providers-celery==3.7.2
apache-airflow-providers-common-io==1.3.2
apache-airflow-providers-common-sql==1.14.2
apache-airflow-providers-fab==1.2.2
apache-airflow-providers-ftp==3.10.0
apache-airflow-providers-http==4.12.0
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-sqlite==3.8.1

### Deployment

Virtualenv installation

### Deployment details

venv setup from python 3.12.4
`pip install "apache-airflow[celery]==2.9.3" --constraint "./constraints-3.8.txt"`

### Anything else?

This is from a completely fresh installation. The issue also appears on an ARM Mac. I don't have the other possible secret backends set up, so I don't know how those behave. I can also understand that this might be a documentation issue and this isn't supported anymore, since the secrets backend would be retrieving secrets for the same file it is configured in itself.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the behavior with the shown airflow.cfg and .env, then inspect the airflow config list/get-value entry points and LocalFilesystemBackend handling. Done means the [database] sql_alchemy_conn_secret setting resolves sql_alchemy_conn instead of returning the literal suffix value and applying the default.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.