Ability to sync airflow variables during deployments
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 484
Description
### Description
The airflow CLI has a set of [variables commands](https://github.com/apache/airflow/blob/main/airflow/cli/commands/variable_command.py). It would be useful to support a command like `airflow variables sync file.json` that
1. adds new variables from the file to airflow's variables
2. updates values of variables that have changed
3. removes variables that are no longer available in the file
### Use case/motivation
We currently manage our airflow variables as an ansible configuration on our team and these variables are version controlled. Different teams will have different solutions to manage airflow variables. If these variables are updated via a continuous deployment (CD) pipeline, every airflow user will have to implement the sync feature.
The feature is fairly common (in our understanding of how people use airflow) and trivial to implement. However, it would help make deployments simpler and quicker to setup for everyone.
The individual building blocks for this change are already available. The (design/philosophical) choice is between making Airflow easier to use and keeping airflow minimalistic i.e. provide the building blocks for users to build on. I'm unsure what the exact design philosophy for Airflow and I'm hoping this ticket can help provide clarity.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reading airflow/cli/commands/variable_command.py and the existing variables commands. Clarify the design and JSON contract before implementation; done means a sync command can add new variables, update changed values, and remove variables absent from the file.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100