Scout reaper: run during planning so a wedged lane recovers within a tick, not a full interval
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.9k
- Forks
- 3.4k
- Avg merge
- 6h 51m
- Merged PRs (30d)
- 232
Description
Background
PR #65028 restored _self_heal_stale_runs to reap orphaned scout runs. The reaper sits on the dispatch path (inside arun_signals_scout), running just before the single-flight guard.
Gap
Because the reap only runs when a child workflow is actually dispatched, an orphaned lane isn't checked until the config is next due — not within the cutoff or the 30-min coordinator tick. Compounding this, _stamp_dispatched_runs advances last_run_at even when the dispatched child later skips on the running guard, and _overdue_seconds then suppresses the lane for its full run_interval_minutes.
Net effect: recovery is bounded by run_interval_minutes, not by STALE_RUN_CUTOFF_S.
- Default 3h scouts: recovers within a couple of hours — fine.
- Daily general scout: up to ~a day of extra suppression after a crash (one missed daily run).
- Pathological max-interval configs: longer.
This is still a bounded recovery (the permanent wedge — the original incident — is fixed), but the daily default lane this PR is meant to protect can still miss a run.
Proposed fix (either or both)
- Run the reaper during planning (
_collect_planned_runsinscout_coordinator.py) so any orphan is reaped within a tick regardless of due-ness. - Do not advance
last_run_atwhen a dispatch resolves to a skip on the running guard, so a wedged lane stays due and re-dispatches next tick instead of being suppressed for a full interval.
Source
Raised by the codex reviewer (P2) on PR #65028: https://github.com/PostHog/posthog/pull/65028#discussion_r3446970733
Also listed as an intended fast-follow in that PR's agent context.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in scout_coordinator.py at _collect_planned_runs, then trace arun_signals_scout, _stamp_dispatched_runs, and _overdue_seconds. Compare the two proposed fixes and verify how planning, the running guard, and last_run_at interact. Done means an orphaned lane is reaped within a coordinator tick and a skipped dispatch does not suppress the lane for a full interval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100