apache / apache/airflow

Partitioned DAG has two Pending Dag Runs with the same partition key

Open
#71,070 0 comments 0 reactions 0 assignees View on GitHub
area:core kind:bug needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Under which category would you file this issue?

Airflow Core

### Apache Airflow version

3.3.0

### What happened and how to reproduce it?

DAG was not triggered due to 2 Pending Dag runs still waiting on different asset events.
Both however have the same Mapped Partition Key and should be considered as one.
Additionally clicking on the received asset events an empty tooltip is shown:

Image

Records in asset_partition_dag_run show two records made with one ms apart. Both are identical except for the partition_date.
One record has it filled in, the other is empty:

Image

### What you think should happen instead?

DAG should be triggered when all 4 events were received.

### Operating System

Ubuntu 22.04.5 LTS

### Deployment

Virtualenv installation

### Apache Airflow Provider(s)

_No response_

### Versions of Apache Airflow Providers

_No response_

### Official Helm Chart version

Not Applicable

### Kubernetes Version

_No response_

### Helm Chart configuration

_No response_

### Docker Image customizations

_No response_

### Anything else?

_No response_

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the partitioned DAG behavior described for Apache Airflow 3.3.0 and inspect the asset_partition_dag_run records, especially the duplicate rows that differ only in partition_date. Trace how the pending runs and received asset events are represented, including the empty event tooltip. Done means identical mapped partition keys are treated as one run, the DAG triggers after all four events, and the tooltip displays the event details.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.