apache / apache/airflow

Auto Refresh Connection Strings to database, to support connection strings with short lived credentials

Open
#55,918 3 comments 0 reactions 0 assignees View on GitHub
area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

This request is to add the ability for airflow to refresh database Connection Strings by re-evaluating the _CMD at a specified refresh interval.

### Use case/motivation

We run Airflow on Kubernetes Clusters on Azure with PostgreSQL as the database backend.

Various CSPs have managed identities, (eg. Azure's workload identity), which can be used to generate access tokens for authenticating against databases like Azure PostgreSQL.

From this discussion - https://github.com/apache/airflow/discussions/43837, I see that we have an option to specify a CMD to execute to obtain the connection string.
While the configuration system supports dynamic CMD evaluation, the database connection string seems to be typically evaluated once during startup and then cached. The CMD would only be re-evaluated during specific reconfiguration scenarios, so it's not guaranteed to be refreshed before tokens expire. So, this is not helpful for cases where the connection strings need to be refreshed periodically as the expiration time is ~ 1 hour.

Restarting all the airflow services frequently is not practical for us, as it increases risks of disruption.

So, one solution may be to bring in changes in the code to refresh/reconfigure connection pool at specified intervals.

I was doing a little research on how this can be done in the current state and this is an option I got - to force re-evaluation of the CMD using "settings.reconfigure_orm()". Though this can be done using a kubectl command like the one below, this will not be executed in the context of the process so I think it wouldn't work:

```
kubectl exec -n airflow-prod airflow-worker-2 -c airflow-worker -- python -c "
import logging
logging.basicConfig(level=logging.INFO)
from airflow import settings
print('Running reconfigure_orm() on worker-2...')
settings.reconfigure_orm()
print('Worker-2 ORM reconfiguration completed successfully!')
"
```

What else can I do to achieve this out of the box, without having to restart the airflow pods?

### Related issues

There is no current issue, but I see a past discussion which is related but not concluded, but the issue I'm talking about is to support short lived tokens - https://github.com/apache/airflow/discussions/43837

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading the current settings.reconfigure_orm() path and how connection-string _CMD values are evaluated and cached. Determine where a refresh interval and connection-pool reconfiguration would fit without restarting Airflow services. Done means short-lived database credentials can refresh automatically without pod restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, kubernetes, postgresql, python
Domain
backend, databases, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.