PM wake chain liveness: turn watchdog + supervised subscription fibers
Open
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 23k
- Forks
- 5.9k
- Avg merge
- 11h 14m
- Merged PRs (30d)
- 357
Description
Symptom
Sometimes a task stage finishing doesn't wake the PM — worse the longer the app has been up. Two liveness defects can wedge the entire wake chain with no recovery short of restart:
- PM turn await has no timeout.
DriverPmAdapter.runTurnblocks onDeferred.await(DriverPmAdapter.ts:713-728) resolved only byturn.completed/turn.abortedwith a matching turnId.session.exited/ error-state events fall through (:581-582) and never fail the prompt;failActivePromptOnBridgeEndonly fires if the whole global event stream ends, which never happens. A hung/zombie PM session holds thePmReEntryQueuesemaphore forever (PmReEntryQueue.ts:91,104) — every later settlement queues behind it, and the reconciliation sweep's redrive also hangs while holdingreconciliationSemaphore(PmRuntime.ts:1822), stalling all future sweeps. - Wake-chain subscription fibers are unsupervised. On stream failure they die silently and are never restarted:
- PmRuntime live subscription logs then terminates permanently (
PmRuntime.ts:2026-2034); its Queue consumer has no defect handler at all (:2062-2066). ProviderRuntimeIngestion.start/CheckpointReactor.start/ProviderCommandReactorfork consumers with no catchCause (ProviderRuntimeIngestion.ts:2036-2051,CheckpointReactor.ts:917-940,ProviderCommandReactor.ts:1213-1214).- Per-instance adapter bridges in
ProviderService.reconcileInstanceSubscriptionshave no error handling and are only re-forked on adapter identity change (ProviderService.ts:336-344).
- PmRuntime live subscription logs then terminates permanently (
Fix plan
- Wrap the PM turn deferred wait in
Effect.timeout; fail the active prompt onsession.exited/error-state runtime events for the PM thread inhandleRuntimeEvent. Downstream hold/redrive semantics already exist. - Add a shared "restart-on-failure with backoff" supervision wrapper (candidate for
packages/shared) applied to every wake-chain subscription listed above.
Acceptance criteria
- A PM provider turn that never completes releases the re-entry semaphore after a bounded timeout and the settlement is redriven.
- Killing/failing any upstream event stream no longer silently stops settlements or checkpointing; the consumer restarts with backoff.
- Metrics/logs make supervision restarts observable.
Size: S-M
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with DriverPmAdapter.ts:713-728 and handleRuntimeEvent around :581-582, then trace the PmReEntryQueue and reconciliation paths. Read the subscription consumers in PmRuntime.ts, ProviderRuntimeIngestion.ts, CheckpointReactor.ts, ProviderCommandReactor.ts, and ProviderService.ts. Done means bounded PM waits release semaphores, failed streams restart with backoff, and restart metrics or logs are observable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100