airflow.cfg can't pull secrets from LocalFilesystemBackend
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Apache Airflow version
2.9.3
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When starting an Airflow instance or checking a property via `airflow config list/get-value` properties that have the `_secret` suffix return their literal value and the main property isn't set at all. The default values instead are applied.
### What you think should happen instead?
According to the docs, the following snippet should retrieve the key `sql_alchemy_conn` from the secret backend (the local filesystem one, in my case) and assign it to the `sql_alchemy_conn` under the `[database]` section.
```
[database]
sql_alchemy_conn_secret = sql_alchemy_conn
```
https://airflow.apache.org/docs/apache-airflow/stable/howto/set-config.html
### How to reproduce
Reproducing requires only configuring the following properties in airflow.cfg and the .env file in the same directory. The secret backend still works from within a DAG, however, it just seems to not be invoked while parsing the airflow.cfg. I tried bringing the [secrets] section to the top of the airflow.cfg, since by default it comes after [database], but that had no effect.
```
[secrets]
backend = airflow.secrets.local_filesystem.LocalFilesystemBackend
backend_kwargs = {"variables_file_path": "./.env"}
[database]
sql_alchemy_conn_secret = sql_alchemy_conn
```
and an .env file (or yaml or json) like this
```
sql_alchemy_conn=postgresql://airflow_user@localhost/airflow_db
```
### Operating System
Arch linux 6.10.4-arch2-1
### Versions of Apache Airflow Providers
apache-airflow-providers-celery==3.7.2
apache-airflow-providers-common-io==1.3.2
apache-airflow-providers-common-sql==1.14.2
apache-airflow-providers-fab==1.2.2
apache-airflow-providers-ftp==3.10.0
apache-airflow-providers-http==4.12.0
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-sqlite==3.8.1
### Deployment
Virtualenv installation
### Deployment details
venv setup from python 3.12.4
`pip install "apache-airflow[celery]==2.9.3" --constraint "./constraints-3.8.txt"`
### Anything else?
This is from a completely fresh installation. The issue also appears on an ARM Mac. I don't have the other possible secret backends set up, so I don't know how those behave. I can also understand that this might be a documentation issue and this isn't supported anymore, since the secrets backend would be retrieving secrets for the same file it is configured in itself.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reproducing the behavior with the shown airflow.cfg and .env, then inspect the airflow config list/get-value entry points and LocalFilesystemBackend handling. Done means the [database] sql_alchemy_conn_secret setting resolves sql_alchemy_conn instead of returning the literal suffix value and applying the default.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100