apache / apache/airflow

Add support to KPO to manage secrets passing Connection-derived credentials

Open
#28,086 15 comments 4 reactions 0 assignees View on GitHub
area:providers good first issue kind:feature provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Add an option to automatically create, map, and manage secrets from Airflow Connection to a Kubernetes Secret for Kubernetes Pod Operator.

### Use case/motivation

One of the important use-cases for Kubernetes Pod Operator is to be able to use credentials passed from Airflow Connections to the POD running the workload.

Some of the users pass the credentials using environment variables, but this is inherently insecure, because environment variables passed to the Pod might be displayed and accessed using various mechanisms and certain scenarios (like failing to create Pod) might reveal it in various logs which are not protected by secret masker.

This is possible even now, but it requires custom Kubernetes Pod Operator which would grab the credential, store it in a secret and mount the secret to the Pod via pod template and delete the secrets after the Pod completes.

This has some drawbacks - it is impossible to attach secret's lifecycle to the Pod lifecycle so managing such secrets (especially deleting them in failure scenarios) has to be done externally (for example by a cleanup script).

There is a "complex-ish" possibilty of managing those secrets by the operator itself. The best idea is to assign a fixed name to the secret (either connection or dag/task based), deleting them and handling potential race scenarios where "deletion" of the secret from one Pod completing would race with creating the secret while starting another Pod.

It would be great to implement such a feature for KPO.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start at KubernetesPodOperator and review how Airflow Connections, Kubernetes Secrets, pod templates, and task completion are currently handled. Define the option's secret creation, mapping, cleanup, and race-safety behavior, then verify that credentials reach the workload without exposing them in logs and that failure scenarios clean up correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.