Add support to KPO to manage secrets passing Connection-derived credentials
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
Add an option to automatically create, map, and manage secrets from Airflow Connection to a Kubernetes Secret for Kubernetes Pod Operator.
### Use case/motivation
One of the important use-cases for Kubernetes Pod Operator is to be able to use credentials passed from Airflow Connections to the POD running the workload.
Some of the users pass the credentials using environment variables, but this is inherently insecure, because environment variables passed to the Pod might be displayed and accessed using various mechanisms and certain scenarios (like failing to create Pod) might reveal it in various logs which are not protected by secret masker.
This is possible even now, but it requires custom Kubernetes Pod Operator which would grab the credential, store it in a secret and mount the secret to the Pod via pod template and delete the secrets after the Pod completes.
This has some drawbacks - it is impossible to attach secret's lifecycle to the Pod lifecycle so managing such secrets (especially deleting them in failure scenarios) has to be done externally (for example by a cleanup script).
There is a "complex-ish" possibilty of managing those secrets by the operator itself. The best idea is to assign a fixed name to the secret (either connection or dag/task based), deleting them and handling potential race scenarios where "deletion" of the secret from one Pod completing would race with creating the secret while starting another Pod.
It would be great to implement such a feature for KPO.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start at KubernetesPodOperator and review how Airflow Connections, Kubernetes Secrets, pod templates, and task completion are currently handled. Define the option's secret creation, mapping, cleanup, and race-safety behavior, then verify that credentials reach the workload without exposing them in logs and that failure scenarios clean up correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100