pingdotgg / pingdotgg/t3code

PM wake chain liveness: turn watchdog + supervised subscription fibers

Open
#7,777 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
23k
Forks
5.9k
Avg merge
11h 14m
Merged PRs (30d)
357

Description

Symptom

Sometimes a task stage finishing doesn't wake the PM — worse the longer the app has been up. Two liveness defects can wedge the entire wake chain with no recovery short of restart:

  1. PM turn await has no timeout. DriverPmAdapter.runTurn blocks on Deferred.await (DriverPmAdapter.ts:713-728) resolved only by turn.completed/turn.aborted with a matching turnId. session.exited / error-state events fall through (:581-582) and never fail the prompt; failActivePromptOnBridgeEnd only fires if the whole global event stream ends, which never happens. A hung/zombie PM session holds the PmReEntryQueue semaphore forever (PmReEntryQueue.ts:91,104) — every later settlement queues behind it, and the reconciliation sweep's redrive also hangs while holding reconciliationSemaphore (PmRuntime.ts:1822), stalling all future sweeps.
  2. Wake-chain subscription fibers are unsupervised. On stream failure they die silently and are never restarted:
    • PmRuntime live subscription logs then terminates permanently (PmRuntime.ts:2026-2034); its Queue consumer has no defect handler at all (:2062-2066).
    • ProviderRuntimeIngestion.start / CheckpointReactor.start / ProviderCommandReactor fork consumers with no catchCause (ProviderRuntimeIngestion.ts:2036-2051, CheckpointReactor.ts:917-940, ProviderCommandReactor.ts:1213-1214).
    • Per-instance adapter bridges in ProviderService.reconcileInstanceSubscriptions have no error handling and are only re-forked on adapter identity change (ProviderService.ts:336-344).

Fix plan

  1. Wrap the PM turn deferred wait in Effect.timeout; fail the active prompt on session.exited/error-state runtime events for the PM thread in handleRuntimeEvent. Downstream hold/redrive semantics already exist.
  2. Add a shared "restart-on-failure with backoff" supervision wrapper (candidate for packages/shared) applied to every wake-chain subscription listed above.

Acceptance criteria

  • A PM provider turn that never completes releases the re-entry semaphore after a bounded timeout and the settlement is redriven.
  • Killing/failing any upstream event stream no longer silently stops settlements or checkpointing; the consumer restarts with backoff.
  • Metrics/logs make supervision restarts observable.

Size: S-M

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with DriverPmAdapter.ts:713-728 and handleRuntimeEvent around :581-582, then trace the PmReEntryQueue and reconciliation paths. Read the subscription consumers in PmRuntime.ts, ProviderRuntimeIngestion.ts, CheckpointReactor.ts, ProviderCommandReactor.ts, and ProviderService.ts. Done means bounded PM waits release semaphores, failed streams restart with backoff, and restart metrics or logs are observable.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, observability
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.