debugging: a step debugger as a first-class surface — pause, inspect, step, and resume; local and durable; humans and agents

Owner direction (2026-08-22): consider interactive, breakpoint-style step debugging as a first-class feature — for humans and agents alike, local and remote, during development, testing, and optionally against in-flight production runs — with the security and performance properties thought through rather than bolted on. This is the design pass, recorded; it folds into the testing charter (#405) beside DST (#477), because a debugger and a schedule explorer are two interfaces to the same seam.

The short answer

Yes, and the repo is unusually well placed — because, as with DST, the prerequisites already exist without being called that. A debugger needs four things: a place to pause, a way to inspect, a way to resume or step, and an answer to "who may do this to which run". All four have landed machinery:

• A place to pause. The durable interpreter already suspends at step boundaries: RunState.StepsBudget decrements per step and the run Continues-As-New when it runs out (engine/workflow.go:419,557) — the examples corpus is even tested with a budget of 1, suspending between every step. A breakpoint is a conditional use of the boundary that already exists; disabled, it is a nil check there, which is the whole performance story.
• A way to choose what advances. The local driver's scheduling decisions are a value — v1.Scheduler (scheduler.go:73), injected, with WrittenOrder as the default and seeded schedules as the DST tier's implementation. An interactive debugger is a third Scheduler: one that asks a person (or an agent) which ready unit advances. That framing buys something rare: a debug session is a hand-driven schedule, so it can be recorded and replayed as a seed the way a DST divergence already is ("every failure is a seed", #477) — step through once, keep the schedule, replay it forever.
• A way to inspect in flight. The durable driver already serves read-only queries on a running workflow: ProgressQuery and StateQuery (engine/progress.go:389,406), and flow watch renders a live view with declared-sensitive outputs redacted behind the shared --reveal-sensitive flag (cmd/flow/watch.go). Inspection is an extension of a surface that exists, not a new one.
• A way to poke a run, gated by policy. Signals are the precedent for "an outside party acts on a run, and the workflow's policy decides who": SignalPolicyCheck, structured claim matching, distinct_from_starter: (proto/flowstate/v1/signal.proto), rehearsal senders refused durably. "Who may debug" is the same question with a stricter answer, and it should reuse the same policy vocabulary rather than invent a fourth claims spelling (the #726 lesson).

The shape, in slices

Slice 1 — local, where the whole value is reachable cheaply. flow test --run <case> --debug and flow run local --debug, breaking at step boundaries in the real local driver: before a step, show the step, its resolved inputs, and the scope; commands are step, continue, until , inspect (evaluate a CEL expression against the current scope — the engine's own evaluator, cost-bounded exactly as everywhere else, DefaultCostLimit), and for flow test, which stub would answer. The virtual clock must treat a paused debugger as a participant that holds time — the participant accounting and the untimed-wait escape (clock.go:189,219) are the exact machinery, and the pause-forever case is the one LeaveClockWhile already documents. Illustrative session, not the landed shape:

$ flow test --run 'high risk' --debug release.test.yaml
break at approval (wait_for_signal: ship-approved, timeout 1h)   t=0s
debug> inspect steps.build.artifact
"checkout-1.2.3"
debug> signals
ship-approved scripted at t=5m {approved: true}
debug> continue
break at route (switch on ${inputs.risk} = "high")               t=5m
debug> step
ship_canary → stub 4 answers {status_code: 200}

Slice 2 — durable, read-mostly first. Pause/resume as a signal the interpreter checks at the step boundary it already visits; inspect as a query handler beside ProgressQuery. Because the boundary check reads durable state, a run can be flipped into debug mode while in flight by signal — the "enable on a live run" case costs nothing extra by construction. Stepping a durable run is exactly resume-with-budget-1, which StepsBudget already implements. What slice 2 does not do: control Temporal's own activity scheduling — the cut #477 draws for DST holds here for the same reason.

Slice 3 — the agent surface. The same verbs over MCP (flowstate_debug_* or one session-scoped tool), because an agent mid-incident wants "pause it, show me the scope, step once" exactly as a human does. The binding constraint is resource holding: a paused local debug session holds a registry and a clock, and flow mcp --serve already bounds one flowstate_test call's time for precisely this reason (cmd/flow/mcpserve.go:127-131) — a debug session gets a lease with a timeout, and an abandoned session expires to resume-and-finish (or fail-safe cancel; question 4). Fail closed on abandonment, never hold forever.

Security properties, decided up front

• Deny by default. A durable run is debuggable only where a policy says so — no policy, no pause, no inspect. Local runs are the author's own process and stay always-on, the flow test trust model.
• The policy reuses the signals vocabulary. Same structured claims, same attested-sender rule; a rehearsal identity is refused durably exactly as scripted signal senders are (signalrehearsal.go). Whether it lives as a debug: sibling of signals: in the Flowfile or as deployment configuration is question 3 — the file spelling makes "who may debug this workflow" reviewable next to "who may signal it", which is where an auditor would look.
• Inspection never resolves a secret. Secrets reach a worker as references and resolve only inside the activity that needs the value (CLAUDE.md's containment rule); a paused run's scope holds references, and inspect renders them as references. Declared-sensitive: values redact behind the same etiquette flow watch and flow get already share. Debug output is a new egress channel for run data and must sit behind the same redaction, not a parallel implementation.
• Every debug action is attributable. Pause, inspect-expression, resume land in the run's visible record (the memo/history surface that already records signal senders), so a debugged production run says it was debugged, by whom, and what they evaluated. An inspection channel that leaves no trace is an exfiltration channel.
• No new wire surface in slice 1. Local debugging adds zero listening anything; durable debugging rides the existing client-to-server RPCs (signal + query), inheriting their auth interceptors unchanged.

Performance

Disabled is the only path production takes by default and it must stay indistinguishable from today: the step boundary already exists (StepsBudget is checked there now), so "no debugger" is one more nil/flag check per step against work the interpreter already does per step. No always-on session server, no instrumentation tax, nothing on the hot path of task execution itself. The costs live only inside an active session (a held registry locally; an extra query handler durably), and the lease bounds them.

What this must not become

A second execution model. The debugger drives the real drivers through seams they already expose — scheduler, step boundary, query, signal — so a debugged run is the run, not a simulation of it, and both-drivers conformance keeps meaning one thing. Any design that forks an interpreter loop for debugging re-creates the two-drivers problem inside one driver, and should be refused on that ground alone.

Questions

1. Greenlight slice 1 (local --debug over the scheduler/boundary seam, virtual-clock-aware)? (Recommended — it is where authors live, it needs no policy work, and it forces the seams into shape before anything durable depends on them.)
2. Record-and-replay from day one? A debug session records its schedule; --seed-style replay reproduces it. (Recommended — it is nearly free under the Scheduler framing and it is what makes a debugging session a test artifact instead of an ephemeral one.)
3. Where does the durable debug policy live — a debug: stanza beside signals: in the Flowfile, deployment config, or both (file declares, deployment may narrow)? (Both recommended, matching how signal policy and deployment policy already layer.)
4. Abandoned durable session disposition: resume-and-finish, or hold-until-timeout-then-resume? (Resume recommended; a run held paused by a vanished debugger is an availability incident.)
5. Schema: the debug session's wire messages (pause/resume/inspect over RPC) are boundary-crossing types and belong in the schema per the standing rule — agreed, or is slice 2 deferred until #923's decision settles the harness's own schema story? (Sequencing question only; the answer to "schema when it crosses" is not in doubt.)

Verified against 13e3229 (origin/main, 2026-08-22). Cross-references: #477 (DST — the shared Scheduler seam and the every-failure-is-a-seed discipline), #405 (testing charter placement), #923 (the harness schema decision slice 2 sequences against), #131 (heartbeats/progress — the observability substrate inspect extends).

Generated by Claude Code

Read the proposed seams in engine/workflow.go, scheduler.go, clock.go, engine/progress.go, cmd/flow/watch.go, and cmd/flow/mcpserve.go, then review related issues #405, #477, #923, and #131. The design pass is complete when the five listed questions have decisions and the scope, policy, schema, abandonment behavior, and record-and-replay expectations are settled.