Auto Refresh Connection Strings to database, to support connection strings with short lived credentials
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
This request is to add the ability for airflow to refresh database Connection Strings by re-evaluating the _CMD at a specified refresh interval.
### Use case/motivation
We run Airflow on Kubernetes Clusters on Azure with PostgreSQL as the database backend.
Various CSPs have managed identities, (eg. Azure's workload identity), which can be used to generate access tokens for authenticating against databases like Azure PostgreSQL.
From this discussion - https://github.com/apache/airflow/discussions/43837, I see that we have an option to specify a CMD to execute to obtain the connection string.
While the configuration system supports dynamic CMD evaluation, the database connection string seems to be typically evaluated once during startup and then cached. The CMD would only be re-evaluated during specific reconfiguration scenarios, so it's not guaranteed to be refreshed before tokens expire. So, this is not helpful for cases where the connection strings need to be refreshed periodically as the expiration time is ~ 1 hour.
Restarting all the airflow services frequently is not practical for us, as it increases risks of disruption.
So, one solution may be to bring in changes in the code to refresh/reconfigure connection pool at specified intervals.
I was doing a little research on how this can be done in the current state and this is an option I got - to force re-evaluation of the CMD using "settings.reconfigure_orm()". Though this can be done using a kubectl command like the one below, this will not be executed in the context of the process so I think it wouldn't work:
```
kubectl exec -n airflow-prod airflow-worker-2 -c airflow-worker -- python -c "
import logging
logging.basicConfig(level=logging.INFO)
from airflow import settings
print('Running reconfigure_orm() on worker-2...')
settings.reconfigure_orm()
print('Worker-2 ORM reconfiguration completed successfully!')
"
```
What else can I do to achieve this out of the box, without having to restart the airflow pods?
### Related issues
There is no current issue, but I see a past discussion which is related but not concluded, but the issue I'm talking about is to support short lived tokens - https://github.com/apache/airflow/discussions/43837
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reading the current settings.reconfigure_orm() path and how connection-string _CMD values are evaluated and cached. Determine where a refresh interval and connection-pool reconfiguration would fit without restarting Airflow services. Done means short-lived database credentials can refresh automatically without pod restarts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, kubernetes, postgresql, python
- Domain
- backend, databases, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100