Add on_catchup_skip_callback for DAGs with catchup=False
- Dominant language
- Python
- Stars
- 46.9k
- Forks
- 17.8k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 483
Description
### Description
When a Dag is configured with `catchup=False` and a scheduler restart (or a DAG re-enable after a pause) leaves a gap between the previous data interval and the current time, the scheduler calls the timetable's `_skip_to_latest()` to advance past the missed intervals. The skip happens silently: no listener event, no callback, and no DagRun row is created for the gap.
This is the right default — `catchup=False` users explicitly opt out of backfill. But operators of business-critical Dags still need to *know* an interval was skipped, distinct from "no run because the schedule didn't fire" and "no run because the task failed". Today the only way to detect a skip is to compare `last_automated_data_interval` against the scheduler's wall clock externally.
Dag-level callbacks exist for terminal DagRun states (`on_success_callback`, `on_failure_callback`) and for SLA misses (`sla_miss_callback`). There is no symmetric hook for "an interval was skipped".
### Use case / motivation
- SLA accounting on data-aware Dags: an analyst needs to know the 2024-01-15 partition will never be filled because the scheduler skipped past it on a Saturday restart.
- Observability dashboards that count Dag runs per window need a counter for skipped intervals, not just `success` / `failed`.
- Compliance / audit Dags need a paper trail for any interval the platform decided not to backfill.
### Proposal
Add an `on_skipped_intervals_callback: Callable[[DAG, list[DataInterval]], None]` parameter to the `DAG` constructor. The scheduler invokes it once per call to `_skip_to_latest` with the list of (start, end) intervals it advanced past. The callback runs in the scheduler the same way `sla_miss_callback` does today; it does not create DagRun rows.
A symmetric listener hook (e.g. `on_intervals_skipped`) could be added alongside the callback so AIP-61 listeners can observe the same event.
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Contributor guide
Research direction
Start at the DAG constructor and the scheduler path that calls the timetable's _skip_to_latest(), then compare how sla_miss_callback is invoked. Determine how skipped DataInterval values are collected and passed to the proposed callback without creating DagRun rows; completion should include coverage for scheduler restarts or DAG re-enabling with catchup=False.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100