apache / apache/airflow

Enhance Variable.set to create versions in GCP Secret Manager when backend is connected

Open
#40,438 0 comments 0 reactions 0 assignees View on GitHub
kind:feature provider:google
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

When Airflow is configured to use GCP Secret Manager as the backend for variable storage, Variable.get() retrieves the latest version of the secret from Secret Manager. However, Variable.set() currently creates a variable in the Airflow database instead of creating a new version in Secret Manager.

This inconsistency can lead to confusion and potential security risks.

I propose enhancing Variable.set() method to add the possibility to create a new version of the secret in Secret Manager when the backend is connected, maintaining consistency with Variable.get() behavior.

### Use case/motivation

- Consistency: Users expect Variable.set() to interact with Secret Manager when it's configured as the backend, just as Variable.get() does.
- Version control: Creating new versions in Secret Manager allows for better tracking and management of variable changes over time.
- Security: Storing variables directly in Secret Manager, rather than the Airflow database, maintains the security benefits of using a dedicated secret management service.

### Related issues

None that I'm aware of.

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with the Variable.set() and Variable.get() entry points and the GCP Secret Manager backend. Trace how the backend is connected and how Secret Manager versions are created, then define the expected behavior for setting variables through it. Add coverage for the relevant variable and GCP backend behavior and verify the existing retrieval path remains consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.