apache / apache/airflow

Problem with Rotation of Fernet Key

Open
#47,963 9 comments 0 reactions 0 assignees View on GitHub
area:helm-chart area:secrets kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Official Helm Chart version

1.15.0 (latest released)

### Apache Airflow version

2.9.3

### Kubernetes Version

1.31.4

### Helm Chart configuration

```yaml
createUserJob:
applyCustomEnv: false
useHelmHooks: false
dags:
gitSync:
branch: main
credentialsSecret: airflow-git-credentials
enabled: true
repo: https://github.com/dacamposol/kg-infra.git
subPath: dags
extraEnv: |
- name: AIRFLOW__API__AUTH_BACKENDS
value: 'airflow.api.auth.backend.basic_auth'
migrateDatabaseJob:
applyCustomEnv: false
jobAnnotations:
'argocd.argoproj.io/hook': Sync
useHelmHooks: false
useStandardNaming: true
```

### Docker Image customizations

_No response_

### What happened

> [!IMPORTANT]
> This is an instance managed via ArgoCD.

1. I created a connection from type HTTP directly from the Web Server UI.
2. I modified a field in the Argo Application in regards of adding the `.extraEnv` to allow REST API calls.
3. I accessed again to the Web Server UI.
4. Cannot open the **Connections** section (all others work fine).

Error in the `webserver` Pod:

```bash
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/mapper.py", line 3702, in _event_on_load
instrumenting_mapper._reconstructor(state.obj())
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/connection.py", line 213, in on_db_load
if self.password:
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/attributes.py", line 606, in __get__
retval = self.descriptor.__get__(instance, owner)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/connection.py", line 340, in get_password
return fernet.decrypt(bytes(self._password, "utf-8")).decode()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cryptography/fernet.py", line 211, in decrypt
raise InvalidToken
cryptography.fernet.InvalidToken
```

When I go to my Argo CD deployment, I can indeed see how for the Airflow application, the `airflow-fernet-key` got recreated at the same time than the sync happened.

I am assuming the Secret got recreated and somehow the rotation process failed, so the **Connections** are now encrypted with the old key, and thus cannot be accessed anymore.

### What you think should happen instead

_No response_

### How to reproduce

1. Have an Airflow Cluster managed by Argo CD.
2. Create a connection in that cluster.
3. Modify settings in the Application, and wait for Sync.
4. Attempt to access again to the Connections in the aforementioned cluster.

### Anything else

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with airflow/models/connection.py and the webserver traceback to trace how Fernet decrypts stored connection passwords. Reproduce the ArgoCD sync using the listed Helm configuration and inspect the airflow-fernet-key Secret before and after synchronization. Done should define and verify expected connection access when the application configuration changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes, python
Domain
devops, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.