apache / apache/airflow

Print error in Airflow UI when a DAG with duplicate DAG_id as another DAG is present

Open
#17,861 3 comments 0 reactions 0 assignees View on GitHub
kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

**Description**

I would like to see an error in Airflow UI (like ``` "Broken DAG: duplicate DAG-id" ```) when a DAG is present that has the same (non-unique) DAG-id as another DAG. Currently, Airflow silently ignores one of these duplicated DAGs.

**Use case / motivation**
For users that either knowingly or unknowingly have multiple DAGs with the same DAG-id (which could happen especially in the case of dynamically generating DAGs), unintended/weird functionality can result as Airflow tries to figure out which DAG is the right one. It is not clear how Airflow will choose, and if that choice can change each time it parses for DAG changes.

**Additional Info**
Currently in Airflow, my understanding is that the DAG file processor creates a new process and a new DagBag per file. As a result each DagBag only parses one DAG-script and thus DAGs with duplicate ids in different scripts are never detected. https://github.com/apache/airflow/blob/main/airflow/dag_processing/processor.py#L613

Additionally, there's already a test and exception (```AirflowDagDuplicatedIdException```) in place for this, but it depends on a single DagBag having multiple DAGs in it, which isn't the case when actually running Airflow, from discussion with @BasPH and @uranusjr.
https://github.com/apache/airflow/blob/2.1.3/airflow/models/dagbag.py#L405-L409
https://github.com/apache/airflow/blob/main/tests/models/test_dagbag.py#L143-L173

**Related Issues**

I did not find other issues related to this, when searching for "duplicate DAG" or "duplicate DAG-id".

Contributor guide

Open the contributing guide

Research direction

Start by reading the DAG file processor around airflow/dag_processing/processor.py:613 and the duplicate-ID handling in airflow/models/dagbag.py. Compare these with tests/models/test_dagbag.py:143-173 to understand the existing exception coverage. Done means duplicate DAG IDs from separate files produce a visible Airflow UI error instead of silently ignoring one DAG.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.