apache / apache/airflow

CronDataIntervalTimetable mid-period start_date delay

Open
#72,427 1 comment 0 reactions 0 assignees View on GitHub
kind:feature needs-triage
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

### Description

When `start_date` is set to a value that falls inside the currently in-progress period (e.g.
deploying a DAG with `start_date` a few hours ago, catchup=False), `CronDataIntervalTimetable`'s
`_align_to_next(earliest)` skips the in-progress period entirely — it only considers periods that
start at or after `start_date` — which pushes the first run out to nearly two full periods later,
not one. Concretely, for `@daily`, deploying at `start_date = now - 3h` today doesn't produce a
first run for today's or tomorrow's data, it produces one for the day-after-tomorrow.

Verified with a script run through `breeze testing core-tests`, calling
`CronDataIntervalTimetable("@daily", timezone="UTC").next_dagrun_info(...)` directly with
`last_automated_data_interval=None`, `restriction.catchup=False`:

| `start_date` | first run's data interval | delay |
|---|---|---|
| `None` | yesterday -> today (already complete) | none -- fires immediately |
| long in the past | yesterday -> today (already complete) | none -- fires immediately |
| `now - 3h` (today) | day-after-tomorrow's full period | ~2 periods |

### Use case/motivation

I don't know if this is intentional (only ever run "clean" full periods that start entirely after
`start_date`, never a period the DAG only existed for part of) or an oversight -- I couldn't find
prior discussion either way. If it's intentional, this issue should probably just be closed. If
not, the `CronTriggerTimetable`-style `_calc_first_run()` -- which chooses between the past-tick
and next-tick based on `run_immediately` / proximity -- is a fix pattern already proven for the
trigger timetables and could plausibly be adapted, though I haven't scoped that out since it
depends on the answer above.

### Related issues

Was previously opened as #72380 (closed to re-open via this form so template labels apply
correctly). That version originally claimed CronDataIntervalTimetable "always waits for the next
boundary" by analogy with CronTriggerTimetable's run_immediately -- verified against actual
runtime behavior and that claim was wrong for the common case (no start_date, or an old one); this
version describes only the narrower, verified edge case.

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)

Contributor guide

Open the contributing guide

Research direction

Start with CronDataIntervalTimetable._align_to_next() and next_dagrun_info(), reproducing the reported @daily case through the direct call described in the issue. Compare the behavior and first-run decision with CronTriggerTimetable._calc_first_run() and determine the intended treatment of a start_date inside the current period; done means the decision is settled and covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.