apache / apache/airflow

Rescheduled sensors can hang before poke while fetching first reschedule start date

Open
#68,010 0 comments 0 reactions 0 assignees View on GitHub
affected_version:3.3 area:scheduler kind:bug
Dominant language
Python
Stars
46.9k
Forks
17.8k
Avg merge
2d 9h
Merged PRs (30d)
472

Description

A rescheduled sensor task can get stuck before entering the sensor `poke()` method when the worker restarts an attempt that already has `TaskReschedule` rows.

Observed behavior:
- The task attempt starts and emits task/listener startup logs.
- The sensor never emits its normal `ExternalTaskSensor` "Poking for Dag ..." log line.
- No external Dag run count request is made for that stuck restart.
- The task remains running until it is killed.
- A retry of the same task instance reaches `poke()` immediately and succeeds.

The source path that matches this behavior is `BaseSensorOperator.execute()` calling `RuntimeTaskInstance.get_first_reschedule_date()` before `poke()`. When `task_reschedule_count > 0`, the Task SDK currently asks the supervisor to fetch the first `TaskReschedule.start_date`, which adds a pre-poke supervisor/API round trip for rescheduled sensors.

Expected behavior:
A rescheduled sensor restart should not need an extra pre-poke supervisor/API request for metadata the API server already knows when it creates the task run context. The worker should receive the first reschedule start date in `TIRunContext` and use it directly, while preserving the existing supervisor request as a compatibility fallback.

Proposed fix:
Add `first_task_reschedule_start_date` to `TIRunContext`, populate it in the Execution API run response when the task instance has reschedules, and have the Task SDK use that value before falling back to `GetTaskRescheduleStartDate`.

This avoids the pre-poke blocking point and keeps older API/server combinations compatible.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.