apache / apache/airflow

DagRun.created_dag_version_id no longer matches its documented contract

Open
#71,453 1 comment 0 reactions 0 assignees View on GitHub
area:core
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

## Background

`created_dag_version_id` (`airflow-core/src/airflow/models/dagrun.py:301-304`) has existed as a column since AIP-65 DAG versioning (#42913/#43735). #49097 added the ORM relationship (`created_dag_version`) on top of it, with a docstring restating what the name already communicated:

> "The id of the dag version column that was in effect at dag run creation time."

and the paired relationship docstring:

> "The dag version that was active when the dag run was created, if available."

Documented or not, the name itself was always the contract — a write-once historical fact, not a live pointer. For a bundle-versioned (pinned) run, the version recorded at creation was — by design — the version that run would run forever. Per a conversation with the original author of AIP-65 DAG versioning: bundle-versioned runs were always meant to execute the version pinned at creation; the way to get "run with the latest code" behavior was to *not* use a versioned bundle at all (unpinned/local bundles always resolve to the latest `DagVersion` — see `DBDagBag._version_from_dag_run`, `airflow-core/src/airflow/models/dagbag.py:210-216`). There was never supposed to be a way to move a pinned run off the version it was created with.

## What changed

Starting with #52177 (introduces the `run_on_latest_version` flag) and #54984 ("Run verify_integrity when cleared to run on latest"), and extended by #59764 and #65835/#66901, `created_dag_version_id` began being **mutated** after creation:

- `_update_dagrun_to_latest_version` — `airflow-core/src/airflow/models/taskinstance.py:344`
- `clear_task_instances(..., run_on_latest_version=True)` — `airflow-core/src/airflow/models/taskinstance.py:474` and `:497`

An open PR, #71425, is currently extending this same mutation to running/queued DagRuns.

None of these renamed or re-documented the field. It still says "recorded at creation" while several code paths now treat it as "the version this run should currently be considered pinned to" — two different, and now conflated, meanings living in one column.

## Why this matters

The mismatch between the documented contract and actual behavior has already produced concrete defects downstream, because other code correctly relied on the *original* contract and nothing flagged that the contract had changed underneath it:

- #71454 — `DagRun.dag_versions` silently drops versions still in use by uncleared task instances after a partial "run on latest version" clear.
- #71455 — `only_new` clear (`_get_new_task_ids`) can report zero new tasks when there genuinely are some, for the same reason.

Both are symptoms, not the root cause. The root cause is that one field is now asked to represent two different things — an immutable historical fact, and a mutable "current pinned version" pointer — with no way for a reader to tell which one they're getting, or whether the run has been through a partial version bump that leaves it internally inconsistent.

## What needs deciding

This is a design question as much as a naming one:

1. **Does "run on latest version" belong on bundle-versioned/pinned runs at all?** The original design intent was that pinning is absolute, and "latest" is achieved by not pinning. If the answer is no, the mutation added since #54984 should be reconsidered/reverted for the pinned case.
2. **If the capability is intentionally kept**, `created_dag_version_id` should stop being overloaded: introduce a genuinely separate, clearly-named mutable field for "the version this pinned run is currently expected to run at" (e.g. `pinned_dag_version_id` or `current_dag_version_id`), keep `created_dag_version_id` strictly immutable as documented, and audit every consumer (`_version_from_dag_run`, `get_dag_for_run_or_latest_version`, `DagRun.dag_versions`, `_get_new_task_ids`, and anything else reading this field) to use the correct one.

Either way, the current state — one field, two incompatible meanings, no rename, no updated docs — should be resolved before more functionality (e.g. #71425) is built on top of it.

---
Drafted-by: Claude Code (Sonnet 5) (no human review before posting)

Contributor guide

Open the contributing guide

Research direction

Start with the documented field and relationship in airflow-core/src/airflow/models/dagrun.py:301-304, then trace the mutation paths in taskinstance.py and version resolution in dagbag.py:210-216. Review consumers including _version_from_dag_run, get_dag_for_run_or_latest_version, DagRun.dag_versions, and _get_new_task_ids. Done means the historical and mutable meanings are resolved without leaving consumers dependent on an ambiguous contract.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.