multi_agent: wait_agent keeps timing out after terminal agent state is observable

What version of the Codex App are you using?

26.820.71523 (VS Code extension host spawning codex.exe), Windows x64

What issue are you seeing?

Summary

With [features] multi_agent = true, a spawned child turn is recorded as completed,
and no explicit error event was recorded in its rollout. A wait_agent call returned
{"timed_out": false} during the child's normal completion at 13:56:55.696 UTC.
After the first completed-state observation at 14:01:24.689 UTC, the terminal state
was exposed by interrupt_agent and list_agents, but subsequent observed
wait_agent calls associated with the agent produced {"timed_out": true} outputs.
The target attribution of send_message calls remains inferential because the
logged arguments contain no recipient ID. The trace does not yet isolate whether
the repeated calls were generated by runtime supervisor logic, model decisions, or
both; the missing safety boundary is observable regardless of that distinction.

In the directly verified 16:04:11–16:59:59 UTC window, 93 send_message calls and
120 wait_agent calls were recorded; all 120 wait_agent outputs were
{"timed_out": true}. Other tool calls were present but are not included in this
213-call send/wait count.

Using the strict post-completion boundary, 136 observed wait_agent calls produced
{"timed_out": true} outputs by 17:11:51.454 UTC. The fixed metering window from
16:02:37.741 to 17:11:51.454 UTC showed a client-reported cumulative input increase
of 44,500,664 tokens across 254 non-zero increments (mean 175,199; median 174,771;
min 157,837; max 188,247). Cached input increased by 43,780,736, approximately
98.4% of the window increase. This is consistent with frequent repeated
large-context model turns; request-level tracing is needed to confirm exact context
resubmission and cache behavior.

The trace shows repeated model/tool round-trips without an effective runtime safety
bound. During collection, the session continued beyond
the fixed measurement cutoff. The affected rollout file was later observed to stop
receiving new data; the cause and exact stop time were not determined by this
read-only check. A related later trace (see Related reproduction below) exhibited
what appears to be the same missing terminal-target / no-progress safeguards in a
different form; this is an observational comparison, not a confirmed shared root
cause.

Timeline (UTC)

1. Aug 31 13:55 — main agent spawns child agent /root/sam3_examples_inventory
   (spawn_agent, fork_turns="all").
2. Aug 31 13:56–14:01 — the child turn is recorded as completed, and no explicit
   error event was recorded in the child rollout (82 lines, ending 13:56:55.043).
   A wait_agent call returned {"timed_out": false} at 13:56:55.696 as the child
   finished. The first observed completed state for the child agent in the main
   session is at 14:01:24.689 UTC, where interrupt_agent exposed the
   already-completed state in its previous_status.completed field (six times by
   17:11:51); list_agents also reports the agent as {"completed": …} (16:11,
   16:52, 17:10).
3. Aug 31 14:03 → 17:11+ — after the first completed-state observation,
   message/wait activity associated with the agent continued. After 14:01:24.689
   UTC, 136 observed wait_agent calls produced "timed_out": true outputs by
   the fixed 17:11:51.454 UTC cutoff. 116 send_message calls were recorded from
   the 14:01 UTC minute onward; recipient attribution remains inferential. The densest
   verified continuous window was 16:04–16:59 (56 min).

The per-turn pattern (excerpt, UTC)

16:04:11 CALL send_message
16:04:18 CALL wait_agent
16:04:28 OUT  wait_agent {"timed_out": true}   (timeout_ms=10000)
16:04:34 CALL wait_agent
16:04:44 OUT  wait_agent {"timed_out": true}
… (repeats; timeout/tool round-trips were followed by metering
   increments of approximately 175K input tokens each)
16:11:13 OUT list_agents     → agent status is completed
… subsequent wait_agent calls continue to time out …
16:50:30 OUT interrupt_agent → previous_status.completed
… subsequent wait_agent calls continue to time out …

During the same period, the UI repeatedly displayed "agent updated" events, making
the stalled session appear active. I have not correlated each UI event to an
individual tool call.

Token accounting (client-reported token_count events, fixed cutoff)

• Window: 16:02:37.741 → 17:11:51.454 UTC (single fixed cutoff, not live-updated)
• Cumulative input counter: 554,135,398 → 598,636,062 = +44,500,664 tokens
• 254 non-zero increments: mean 175,199; median 174,771; min 157,837; max 188,247
• Cached input rose by 43,780,736 tokens ≈ 98.4% of the window increase
• These are consistent with repeated large-context model turns, but request-level
  tracing would be needed to confirm exact context resubmission and cache behavior
• The session-level cumulative counter also contains earlier (pre-incident) usage;
  only the window increments above are attributed to this behavior

Diagnostics that may help locate the state-propagation issue

• In the local state databases, the child thread has a completed turn while the
  corresponding thread_spawn_edges row still has status open. I have not
  confirmed whether open means an active child or merely an unclosed parent-child
  relationship. The discrepancy between the turn state and the edge state is a
  candidate area for investigating why list_agents/interrupt_agent expose the
  terminal state while wait_agent continues to time out; its causal role is
  unconfirmed.
• logs_2.sqlite recorded four ERROR-level entries from
  codex_core::tools::router (core/src/tools/router.rs:323) around
  13:55–14:02 UTC, at least one involving send_message. Not established as causal;
  sanitized excerpts available on request (encrypted payloads and user file contents
  removed).

Recurrence

A related long-running polling pattern appeared earlier: on Aug 29, 17:46–21:05
UTC (3h19m), the main session polled list_agents roughly every 20 seconds
(618 responses; median interval 19.0s) with no observed deadline or budget stopping
it. I have not established that the earlier agent was already completed, or that the
earlier sequence shares the same lifecycle root cause.

Related reproduction (later trace)

In a separate session on Sep 1, a child was interrupted approximately three seconds
after spawn_agent (turn_aborted was recorded and the child produced no normal
function-call activity). The parent then continued issuing send_message calls.
Unlike the main trace, the tool arguments explicitly contained target: "/root";
whether /root was the source agent itself cannot be established from the persisted
source identity alone. This is therefore evidence of a permitted root-target message
and a strong lead for missing self-target validation, but not yet proof of an A→A
delivery loop.

The session issued 1,615 send_message calls in total; one occurred 28 ms before
the first token_count event used below, so 1,614 fell within the metering window.
The send intervals had a median of approximately 6.7 seconds and a minimum of 2.2
seconds (the maximum gap was ~16 minutes, so the sending was not uniformly
high-frequency). From the first token_count event at 10:13:26.108 UTC (24,605)
to the last at 14:38:42.185 UTC (103,229,651), the client-reported cumulative input
counter increased by 103,205,046 tokens. Note that the send-call count
(function-call timestamps) and the token-counter delta (token_count event
endpoints) are different event types and are not expected to match one-to-one.
These observations do not by themselves
establish whether the repetition was generated by runtime orchestration, model
decisions, message replay, or some combination; they demonstrate that no effective
no-progress safety bound stopped the activity during the observed period.

Expected behavior

1. Sticky, authoritative terminal state: once an agent reaches a terminal state,
   wait_agent should consult the authoritative lifecycle state and return that
   terminal state immediately and idempotently. A wait_agent call for an
   already-completed generation must not wait for a new event; it should return the
   persisted terminal result. If a new turn is intended, it must use an explicit new
   generation/operation ID — a completed generation must not be reopened merely
   because send_message is issued against it. wait_agent, list_agents, and
   interrupt_agent should use one authoritative state model or provide equivalent
   versioned/linearizable lifecycle semantics. In this trace, list_agents reported
   the agent as completed and interrupt_agent exposed
   previous_status.completed, while the observed wait_agent calls after
   14:01:24.689 UTC continued to time out.
2. Explicit semantics for messaging terminal agents: if the orchestration is
   messaging an already-terminal agent, the runtime should either reject the
   operation with an explicit terminal-state error or create an explicit new
   lifecycle/generation. Because the logged send_message arguments do not include
   a recipient identifier in the main trace, target attribution remains inferential.
   The related trace below additionally suggests the runtime should reject
   self-targeted messages — compared using internal agent/generation identifiers,
   not display names such as /root — with an explicit error, and keep the
   user-visible commentary channel separate from internal send_message.
3. Runtime-level waiting/backoff: in this trace, repeated
   wait_agent(timeout_ms=10000) calls whose outputs were "timed_out": true were
   followed by new model turns reporting approximately 175K input tokens each,
   predominantly as cached input. Keeping the polling/backoff loop below the model
   layer would avoid repeatedly invoking the model solely to continue waiting.
4. No-progress circuit breaker / budget: stop the loop based on multiple signals
   (consecutive timeouts, total wall-clock deadline, duration without any agent-state
   change, cumulative input-token growth, repeated identical tool sequences), then
   surface options to the user such as: use completed result (when the final message
   is retrievable) / restart agent / cancel / export diagnostics.
5. Honest UI state: distinguish "actively working" from "waiting on a completed
   agent" — repeated "agent updated" events made the stalled loop look like progress.

Notes / what I have not verified

• The send_message message bodies are encrypted in the rollout logs, and the
  recipient agent identifier is not present in the logged arguments; target
  attribution to the completed agent is inferred from the surrounding orchestration
  context (interleaved wait_agent/list_agents/interrupt_agent calls referencing
  it).
• I have not confirmed at source level whether send_message to a completed agent
  errors internally; the observable behavior is that subsequent waits timed out even
  after the completed state had become visible, while the terminal state was
  exposed via interrupt_agent and list_agents.
• The trace contains one normal-completion wait_agent response with
  "timed_out": false before the later completed-state observation. This shows the
  behavior is not an unconditional timeout on every path, but it does not establish
  the intended semantics of waiting on an already-terminal generation.
• The meaning of the open spawn-edge status (active child vs. unclosed
  relationship) is not confirmed.
• Token counters are client-reported metering values; billing impact of cached vs
  uncached input is not addressed here.
• All counts use the fixed cutoff 17:11:51 UTC; the strict post-completion boundary
  is 14:01:24.689 UTC (first completed-state observation). For reproducibility: a
  broader count starting at the 14:01:00 UTC minute and using output timestamps
  yields 138 timeouts; that broader count is not the strict post-completion count.

Related issues

• #39854 (abnormal ~678M token consumption via repeated multi-agent polling)
• #24951 (wait_agent/spawn_agent can block for hours; timeout not enforced)
• #41875 (align wait_agent default timeout with prompt-cache TTL)
• #23700 (stale Codex subagents)
• #37299 (wait/status orchestration re-meters full cached context every 10–30s)

Happy to provide additional sanitized excerpts (encrypted payloads and user file
contents removed) if useful.

Start by tracing the wait_agent, list_agents, and interrupt_agent lifecycle paths and compare them with the completed turn and open thread_spawn_edges state. Inspect the router error location at core/src/tools/router.rs:323 and reproduce the terminal-agent sequence if possible. Done should include an agreed, tested boundary for terminal agents and bounded waiting, but the issue does not identify specific tests or files beyond the router location.