Print error in Airflow UI when a DAG with duplicate DAG_id as another DAG is present
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
**Description**
I would like to see an error in Airflow UI (like ``` "Broken DAG: duplicate DAG-id" ```) when a DAG is present that has the same (non-unique) DAG-id as another DAG. Currently, Airflow silently ignores one of these duplicated DAGs.
**Use case / motivation**
For users that either knowingly or unknowingly have multiple DAGs with the same DAG-id (which could happen especially in the case of dynamically generating DAGs), unintended/weird functionality can result as Airflow tries to figure out which DAG is the right one. It is not clear how Airflow will choose, and if that choice can change each time it parses for DAG changes.
**Additional Info**
Currently in Airflow, my understanding is that the DAG file processor creates a new process and a new DagBag per file. As a result each DagBag only parses one DAG-script and thus DAGs with duplicate ids in different scripts are never detected. https://github.com/apache/airflow/blob/main/airflow/dag_processing/processor.py#L613
Additionally, there's already a test and exception (```AirflowDagDuplicatedIdException```) in place for this, but it depends on a single DagBag having multiple DAGs in it, which isn't the case when actually running Airflow, from discussion with @BasPH and @uranusjr.
https://github.com/apache/airflow/blob/2.1.3/airflow/models/dagbag.py#L405-L409
https://github.com/apache/airflow/blob/main/tests/models/test_dagbag.py#L143-L173
**Related Issues**
I did not find other issues related to this, when searching for "duplicate DAG" or "duplicate DAG-id".
Contributor guide
Research direction
Start by reading the DAG file processor around airflow/dag_processing/processor.py:613 and the duplicate-ID handling in airflow/models/dagbag.py. Compare these with tests/models/test_dagbag.py:143-173 to understand the existing exception coverage. Done means duplicate DAG IDs from separate files produce a visible Airflow UI error instead of silently ignoring one DAG.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100