Scout reaper: cancel the orphan's sandbox workflow, not just mark TaskRun failed

Background

PR #65028 restored _self_heal_stale_runs so an orphaned scout run (a TaskRun frozen at QUEUED/IN_PROGRESS because the worker/sandbox died before writing its terminal status) no longer wedges a (team, skill) lane forever. The reaper marks the orphan's TaskRun FAILED on the dispatch path, before the single-flight guard, so the lane recovers.

Gap

The reaper only flips the TaskRun row to FAILED via mark_failed(). The scout activity (arun_signals_scout) and the Tasks ProcessTaskWorkflow that drives the sandbox are separate Temporal lifecycles. The reaper never signals ProcessTaskWorkflow.complete_task (what MultiTurnSession.end() does) or cancels the workflow handle — so if that workflow/sandbox is still alive, it is not actively torn down by the reap.

In the stale-worker scenario this leaves a window where the old sandbox could still emit findings against the dead run, or write a late terminal status, while a fresh run for the same lane has already started — i.e. duplicate/corrupted scout output rather than just an unwedged lane.

Why it's bounded today (not a wedge)

• Background runs (origin_product=SIGNALS_SCOUT) drive a ProcessTaskWorkflow that self-terminates on INACTIVITY_TIMEOUT_DEFAULT_SECONDS = 30 min idle.
• The reaper cutoff is STALE_RUN_CUTOFF_S = 2 × WORKFLOW_HARD_CEILING_S = 32 min.
• Once the scout activity dies it stops sending send_followup_message, so the agent finishes at most its current turn and then idles into that 30-min timeout — usually self-cleaning right around when the reaper fires.
• A late terminal overwrite cannot re-wedge the lane, since _has_running_run only blocks on QUEUED/IN_PROGRESS.

So the exposure is a ≤1-turn emit/overwrite window in a rare timing overlap, not a permanent failure.

Proposed fix

When reaping an orphan, actively tear down its sandbox workflow before clearing the lane: signal ProcessTaskWorkflow.complete_task (or cancel the workflow handle) for the orphan's task_run, rather than relying on the inactivity timeout to converge. Note this means driving a Temporal signal from the currently-sync _self_heal_stale_runs path, so the cleanup needs an async-safe entry point.

Source

Raised by the codex reviewer (P1) on PR #65028: https://github.com/PostHog/posthog/pull/65028#discussion_r3446970728

Trace _self_heal_stale_runs and mark_failed alongside the separate arun_signals_scout and ProcessTaskWorkflow lifecycles. Read how MultiTurnSession.end() invokes ProcessTaskWorkflow.complete_task, then determine an async-safe cleanup path for the synchronous reaper. Done means an orphaned task run's sandbox workflow is actively torn down before the lane is cleared, preventing late output or status writes.