Enhancement: Support Dynamic dag_id Resolution in TriggerDagRunOperator for Clickable Links
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Body
Currently, when using TriggerDagRunOperator with a dynamically determined dag_id (e.g., from XCom or templated fields), the Airflow web UI cannot generate a proper link to the triggered DAG at parse time. This results in a broken or invalid link (e.g., {{ task_instance.xcom_pull(...) }}) or a “DAG not found” error, even though the triggered DAG actually runs successfully.
Use Case / Motivation
We have pipelines that need to trigger different DAGs dynamically, depending on runtime conditions (e.g., data or metadata that arrives in real-time).
While the operator itself correctly triggers the desired DAG, the UI link to the triggered DAG is not usable.
Users rely on the UI link to quickly navigate to the triggered DAG’s run details, so a broken link complicates monitoring and debugging.
Proposed Solution
Allow the Airflow UI to update or determine the dag_id link after the task is executed, possibly by reading from XCom or storing the resolved dag_id in the metadata DB.
Provide a mechanism (e.g., an extra link or a new operator parameter) to supply the final dag_id to the UI once it’s known at runtime.
Alternatively, enhance TriggerDagRunOperator to set the correct link in the UI if dag_id is determined during the operator’s execution.
```
def decide_which_dag(**kwargs):
# Some dynamic logic here
return "some_dynamic_dag_id"
trigger_dynamic = TriggerDagRunOperator(
task_id="trigger_dynamic_dag",
trigger_dag_id="{{ python_callable_returned_id }}", # or from XCom
wait_for_completion=True,
dag=dag,
)
```
in the above scenario, the DAG is triggered correctly, but the link displayed in the UI is not valid.
Contributor guide
Research direction
Start with TriggerDagRunOperator and the Airflow web UI link behavior described in the issue. Trace how the dynamically templated dag_id is resolved and whether XCom or the metadata DB can provide it after execution; done means the UI link reaches the correct triggered DAG run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, web-dev
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100