apache / apache/airflow

Add on_catchup_skip_callback for DAGs with catchup=False

Open
#66,791 1 comment 0 reactions 0 assignees View on GitHub
area:core area:scheduler kind:feature
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 10h
Merged PRs (30d)
483

Description

### Description

When a Dag is configured with `catchup=False` and a scheduler restart (or a DAG re-enable after a pause) leaves a gap between the previous data interval and the current time, the scheduler calls the timetable's `_skip_to_latest()` to advance past the missed intervals. The skip happens silently: no listener event, no callback, and no DagRun row is created for the gap.

This is the right default — `catchup=False` users explicitly opt out of backfill. But operators of business-critical Dags still need to *know* an interval was skipped, distinct from "no run because the schedule didn't fire" and "no run because the task failed". Today the only way to detect a skip is to compare `last_automated_data_interval` against the scheduler's wall clock externally.

Dag-level callbacks exist for terminal DagRun states (`on_success_callback`, `on_failure_callback`) and for SLA misses (`sla_miss_callback`). There is no symmetric hook for "an interval was skipped".

### Use case / motivation

- SLA accounting on data-aware Dags: an analyst needs to know the 2024-01-15 partition will never be filled because the scheduler skipped past it on a Saturday restart.
- Observability dashboards that count Dag runs per window need a counter for skipped intervals, not just `success` / `failed`.
- Compliance / audit Dags need a paper trail for any interval the platform decided not to backfill.

### Proposal

Add an `on_skipped_intervals_callback: Callable[[DAG, list[DataInterval]], None]` parameter to the `DAG` constructor. The scheduler invokes it once per call to `_skip_to_latest` with the list of (start, end) intervals it advanced past. The callback runs in the scheduler the same way `sla_miss_callback` does today; it does not create DagRun rows.

A symmetric listener hook (e.g. `on_intervals_skipped`) could be added alongside the callback so AIP-61 listeners can observe the same event.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start at the DAG constructor and the scheduler path that calls the timetable's _skip_to_latest(), then compare how sla_miss_callback is invoked. Determine how skipped DataInterval values are collected and passed to the proposed callback without creating DagRun rows; completion should include coverage for scheduler restarts or DAG re-enabling with catchup=False.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.