Ability to annotate DAG's source (repo, tag, commit, etc)
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
OpenLineage integration can provide information which DAGs and Task were executed, that datasets were read or written, and so on. Also there are a lot of facets containing all information about DAG and Tasks, e.g. scheduled time, owner, tags and so on.
One of facets OpenLineage have is [SourceCodeLocation](https://openlineage.io/docs/spec/facets/job-facets/source-code-location/) which allows users to see the repo DAG source code was originated from, as well as branch name, commit, tag and so on.
But there is no way to add annotations to DAG which allow to collect this information and send it as some facet. Probably something like this could be implemented:
```python
from airflow.sdk import DAG, SourceCodeLocation
dag = DAG(
dag_id="my_dag",
source_code_location=SourceCodeLocation(
repo_url="http://github.com/user/repo",
branch="main",
path="/some/dag.py",
version="6f1cab2f"
),
)
```
### Use case/motivation
My use case is:
* There is a repo with ETL scripts and DAG .py files. Multiple users work with code within this repo, and create branches with scripts/DAGs updates.
* Each branch (e.g. `bugfix/123` or `feature/123` branch) is deployed to Airflow. To prevent overriding working files of different users, `/opt/airflow/dags` folder have a structure `{repo_name}/{branch_name}/my_dag.py`.
* For Airflow on host, branch can be cloned using `git clone` or `rsync`.
* For Airflow in Kubernetes this is done via custom sidecar which clones every branch within a repo to create this directory structure.
* After branch is merged, directory containing DAGs for this branch is removed from Airflow.
I'm not using `gitsync` sidecar because it clones only one specific branch (e.g. `main`), and cannot be configured without redeploying Airflow workers & dag processor.
Same for `GitDagBundle` in Airflow 3.x - bundles are set up using airflow.cfg, not via API, and changing this config require restarting DagProcessor.
Also OpenLineage integration for Airflow is set up for the entire Airflow instance, not for each DAG specifically. But there can be DAGs from multiple repos, branches or commits within the same branch, and this cannot be passed to OpenLineage via config https://github.com/OpenLineage/OpenLineage/issues/3745.
It is possible to parse all this information from `.git` folder content https://github.com/OpenLineage/OpenLineage/issues/3746, but the precense of this folder actually depends on the way Airflow DAG are reployed. Also, having `.git` folder may be a security issue in some usecases (e.g. public SAAS instance with private Git repos).
That's why here I mention here only some kind of DAG object _annotation_/option.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start by reviewing the airflow.sdk.DAG entry point and how the Airflow OpenLineage integration builds facets. Compare the proposed SourceCodeLocation annotation with the OpenLineage SourceCodeLocation specification, then define how repo, branch, path, tag, and commit values would reach emitted events and how the behavior should be tested.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100