apache / apache/airflow

DagRun.dag_versions silently drops versions after a partial "run on latest version" clear

Open
#71,454 1 comment 0 reactions 0 assignees View on GitHub
area:core area:dag-bundles kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

## Problem

`DagRun.dag_versions` (`airflow-core/src/airflow/models/dagrun.py:515-530`) has a shortcut for bundle-versioned (pinned) runs:

```python
@property
def dag_versions(self) -> list[DagVersion]:
"""Return the DAG versions associated with the TIs of this DagRun."""
# when the dag is in a versioned bundle, we keep the dag version fixed
if self.bundle_version:
return [self.created_dag_version] if self.created_dag_version is not None else []
...
```

This assumes a pinned run's task instances can only ever be on the single version recorded in `created_dag_version_id`. That was true when `created_dag_version_id` was immutable (set once at DagRun creation, per #49097). It is no longer true: `clear_task_instances(..., run_on_latest_version=True)` (added in #54984, extended in #65835/#66901) bumps `created_dag_version_id` to the latest version, but only bumps the `dag_version_id` of the *cleared* task instances — uncleared task instances explicitly keep their old version (see the comment at `airflow-core/src/airflow/models/taskinstance.py:477`: "Only cleared TIs get latest dag_version_id above; do not rewrite others.").

## Repro

1. Create a pinned (bundle-versioned) DagRun with tasks A and B, both created against version V1.
2. Deploy a new version V2 (no task additions/removals needed).
3. Clear only task A with `run_on_latest_version=True`. Now: `dag_run.created_dag_version_id == V2`, task A's TI is on V2, task B's TI is still on V1.
4. Read `dag_run.dag_versions` (or view the Grid UI, which reads it directly at `airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py:353`).

## Expected

`[V1, V2]` — the run genuinely has task instances on both versions.

## Actual

`[V2]` — task B's actual version (V1) is silently dropped, because the property trusts `created_dag_version_id` alone instead of the real per-TI versions once bundle-version pinning is set.

## Impact

The Grid UI's "Dag versions" indicator for a run can show a single version chip while some of that run's tasks are still genuinely executing under an older version — misleading when debugging "what code actually ran here."

## Suggested fix

Don't special-case bundle-versioned runs in `dag_versions`; compute it the same way as unpinned runs (union of `_ti_dag_versions`/`_tih_dag_versions`), or explicitly detect and represent the mixed-version case.

## Context

This is one concrete consequence of `created_dag_version_id`'s contract having drifted from "the version recorded at DagRun creation" (its documented meaning) to "the version this run should currently run at" (its behavior since #54984). See the companion issue tracking that broader contract problem: #71453

---
Drafted-by: Claude Code (Sonnet 5) (no human review before posting)

Contributor guide

Open the contributing guide

Research direction

Start in airflow-core/src/airflow/models/dagrun.py:515-530 and compare the bundle-version shortcut with the unpinned _ti_dag_versions/_tih_dag_versions path. Read the version-update behavior around airflow-core/src/airflow/models/taskinstance.py:477 and inspect the Grid UI use at airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py:353. Done means dag_versions includes every version represented by the run's task instances, including a mixed V1/V2 run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.