CronDataIntervalTimetable mid-period start_date delay
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 472
Description
### Description
When `start_date` is set to a value that falls inside the currently in-progress period (e.g.
deploying a DAG with `start_date` a few hours ago, catchup=False), `CronDataIntervalTimetable`'s
`_align_to_next(earliest)` skips the in-progress period entirely — it only considers periods that
start at or after `start_date` — which pushes the first run out to nearly two full periods later,
not one. Concretely, for `@daily`, deploying at `start_date = now - 3h` today doesn't produce a
first run for today's or tomorrow's data, it produces one for the day-after-tomorrow.
Verified with a script run through `breeze testing core-tests`, calling
`CronDataIntervalTimetable("@daily", timezone="UTC").next_dagrun_info(...)` directly with
`last_automated_data_interval=None`, `restriction.catchup=False`:
| `start_date` | first run's data interval | delay |
|---|---|---|
| `None` | yesterday -> today (already complete) | none -- fires immediately |
| long in the past | yesterday -> today (already complete) | none -- fires immediately |
| `now - 3h` (today) | day-after-tomorrow's full period | ~2 periods |
### Use case/motivation
I don't know if this is intentional (only ever run "clean" full periods that start entirely after
`start_date`, never a period the DAG only existed for part of) or an oversight -- I couldn't find
prior discussion either way. If it's intentional, this issue should probably just be closed. If
not, the `CronTriggerTimetable`-style `_calc_first_run()` -- which chooses between the past-tick
and next-tick based on `run_immediately` / proximity -- is a fix pattern already proven for the
trigger timetables and could plausibly be adapted, though I haven't scoped that out since it
depends on the answer above.
### Related issues
Was previously opened as #72380 (closed to re-open via this form so template labels apply
correctly). That version originally claimed CronDataIntervalTimetable "always waits for the next
boundary" by analogy with CronTriggerTimetable's run_immediately -- verified against actual
runtime behavior and that claim was wrong for the common case (no start_date, or an old one); this
version describes only the narrower, verified edge case.
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
Contributor guide
Research direction
Start with CronDataIntervalTimetable._align_to_next() and next_dagrun_info(), reproducing the reported @daily case through the direct call described in the issue. Compare the behavior and first-run decision with CronTriggerTimetable._calc_first_run() and determine the intended treatment of a start_date inside the current period; done means the decision is settled and covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100