apache / apache/airflow

Add a CRD-sync on k8s

Open
#19,250 4 comments 5 reactions 0 assignees View on GitHub
kind:feature provider:cncf-kubernetes
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

Airflow use `git-sync` to pull DAGs from git repository, I'm thinking about doing it in a more cloud native way.
So I thought of using k8s `custom resource definition` to do this.
We can migrate DAG, files and other contents to `custom resource definition`. Based on the operator by a container, we can instantly synchronize them to each pod of Airflow.

### Use case/motivation

I have created a project on GitHub to implement this idea https://github.com/cdmikechen/airflow-dag-operator.
It will start a synchronization service with [Quarkus Operator](https://github.com/quarkiverse/quarkus-operator-sdk) on each airflow pod to synchronize the DAG/files into the DAG folder.

### Related issues

_No response_

### Are you willing to submit a PR?

- [X] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

The issue names no Airflow files, tests, or entry points; start by reviewing the linked airflow-dag-operator project and Airflow's existing Kubernetes and git-sync integration. Define the CRD, synchronization behavior, and Airflow deployment changes needed to sync DAGs and other files to each pod, with tests and documentation demonstrating the completed workflow.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, python
Domain
data-engineering, devops, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.