apache / apache/airflow

Ability to sync airflow variables during deployments

Open
#31,875 9 comments 1 reaction 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 7h
Merged PRs (30d)
484

Description

### Description

The airflow CLI has a set of [variables commands](https://github.com/apache/airflow/blob/main/airflow/cli/commands/variable_command.py). It would be useful to support a command like `airflow variables sync file.json` that

1. adds new variables from the file to airflow's variables
2. updates values of variables that have changed
3. removes variables that are no longer available in the file

### Use case/motivation

We currently manage our airflow variables as an ansible configuration on our team and these variables are version controlled. Different teams will have different solutions to manage airflow variables. If these variables are updated via a continuous deployment (CD) pipeline, every airflow user will have to implement the sync feature.

The feature is fairly common (in our understanding of how people use airflow) and trivial to implement. However, it would help make deployments simpler and quicker to setup for everyone.

The individual building blocks for this change are already available. The (design/philosophical) choice is between making Airflow easier to use and keeping airflow minimalistic i.e. provide the building blocks for users to build on. I'm unsure what the exact design philosophy for Airflow and I'm hoping this ticket can help provide clarity.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reading airflow/cli/commands/variable_command.py and the existing variables commands. Clarify the design and JSON contract before implementation; done means a sync command can add new variables, update changed values, and remove variables absent from the file.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.