apache / apache/airflow

A way to switch create_cron_data_interval=true back to 2.x run_id logic

Open
#71,187 2 comments 0 reactions 0 assignees View on GitHub
area:core kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

[apache/airflow#59115](https://github.com/apache/airflow/pull/59115) (merged in 3.2.0, as part of AIP-76
partition scheduling support) changed how `run_id` is generated for scheduled DAG runs. This is documented
plainly in [apache/airflow#65688](https://github.com/apache/airflow/pull/65688):

> Airflow 3.2.0 (related PR: #59115) changed how run_id is generated for scheduled DAG runs:
> - 3.1.x: `run_id = scheduled__`
> - 3.2.0: `run_id = scheduled__`

This is confirmed intentional, not a bug, see the base `Timetable.generate_run_id()` implementation:
https://github.com/apache/airflow/blob/3.3.0/airflow-core/src/airflow/timetables/base.py#L460-L474

The issue: `AIRFLOW__SCHEDULER__CREATE_CRON_DATA_INTERVALS` exists specifically to let cron-scheduled DAGs
opt back into Airflow-2-style semantics (logical_date == interval start, run_after offset). It restores
`logical_date`/data-interval behavior, but does **not** cover `run_id` generation, `CronDataIntervalTimetable`
(the class this setting selects) doesn't override `generate_run_id()`:
https://github.com/apache/airflow/blob/3.3.0/airflow-core/src/airflow/timetables/interval.py#L133

So users who explicitly opt into `CREATE_CRON_DATA_INTERVALS=True` for full Airflow-2 semantics still end up
with `run_id` decoupled from `logical_date`, a partial, easy-to-miss gap in what that setting promises.

Related, currently open: [apache/airflow#70167](https://github.com/apache/airflow/pull/70167) is updating
`dag-run.rst` to document the new default `run_id`/`logical_date` behavior, confirming this is the accepted,
documented state going forward, but doesn't address the `CREATE_CRON_DATA_INTERVALS` coverage gap raised here.

What should happen instead:
Two possible directions, in rough order of scope:

1. Extend `CREATE_CRON_DATA_INTERVALS` (or a new, separate flag) to also anchor `run_id` to the data
interval start when enabled.
2. A more general mechanism e.g. a `dag_run_policy`-style cluster policy hook, that would let users
customize `run_id` generation (and potentially other DagRun attributes) without a purpose-built config
flag. This was suggested informally by an Airflow core maintainer as possibly the cleaner long-term
design, though not something planned near-term.

Either would close the gap for deployments that rely on `CREATE_CRON_DATA_INTERVALS` for backward
compatibility during 2→3 migrations.

Workaround:
A custom `Timetable` subclass overriding `generate_run_id()` to key off `data_interval.start` works today
and requires no core changes:

​```python
from airflow.timetables.interval import CronDataIntervalTimetable

class LogicalDateRunIdTimetable(CronDataIntervalTimetable):
def generate_run_id(self, *, run_type, run_after, data_interval=None, **extra) -> str:
if data_interval is not None:
return run_type.generate_run_id(suffix=data_interval.start.isoformat())
return super().generate_run_id(
run_type=run_type, run_after=run_after, data_interval=data_interval, **extra
)
​```

This is DAG-level, not deployment-wide, so it doesn't fully substitute for a config-level fix for users
with many DAGs.

### Use case/motivation

Airflow deployments migrating from 2 to 3 that rely on `CREATE_CRON_DATA_INTERVALS` for backward
compatibility expect that setting to fully restore Airflow-2-style semantics. Currently `run_id` silently
falls outside that guarantee, which can break anything downstream that parses `run_id` and assumes it
reflects `logical_date`, without any warning that the setting's coverage is partial.

### Related issues

_No response_

### Are you willing to submit a PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with airflow-core/src/airflow/timetables/base.py and airflow-core/src/airflow/timetables/interval.py, especially generate_run_id() and CronDataIntervalTimetable, then trace CREATE_CRON_DATA_INTERVALS. Define whether the existing setting or a new mechanism should control run_id generation, and document and test the chosen behavior for Airflow-2-compatible scheduling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.