[Bug]: Wedged Codex rate-limit app-server child blocks the runtime handshake indefinitely (runtimeReachable stays false for 31 min; killing the child recovers without restart)

Operating system

Ubuntu 26.04 LTS, x86_64 (headless orca serve remote runtime host, 8 vCPU / 15 GiB RAM)

Orca version

1.4.183 (Linux .deb, orca-server.service running orca-ide serve --port 6768)

Details

Short summary

A hidden Codex rate-limit subprocess spawned by the Orca main process wedged immediately after launch and blocked the runtime handshake for 31 minutes. orca status reported runtimeState: starting / runtimeReachable: false the entire time, while the main process stayed alive and burned CPU. Killing that one child process (plus an unrelated runaway search) restored the runtime without restarting Orca.

The subprocess is the same class described in #11752 (rate-limit-pty-cwd), but the failure mode here is different: it is not a proxy/timeout problem, it is a wedged child that the runtime start path appears to wait on indefinitely.

Timeline (UTC, from main.trace.ndjson and /proc)

Time	Event
07:25:47	Orca main (pid 1725) spawns child: codex -s read-only -a untrusted app-server, cwd ~/.config/orca/rate-limit-pty-cwd
07:25:50	main.trace.ndjson stops advancing — 3 seconds after the spawn
07:25–07:56	orca status reports runtimeState: starting, runtimeReachable: false, runtimeId: none continuously
07:53	Wedged child killed (SIGTERM ignored, SIGKILL required)
07:56	Runtime reaches ready / runtimeReachable: true, new runtime id — no Orca restart performed

Observed state of the wedged child

pid 835692  ppid 1725 (orca-ide --serve)
/home/dev/.local/bin/codex -s read-only -a untrusted app-server
cwd: ~/.config/orca/rate-limit-pty-cwd
%CPU: 25    RSS: 13.7 MB    elapsed: 27m
disk read rate: ~155 MB/s sustained

The process held only ~14 MB RSS but was spinning: 25% CPU and ~155 MB/s of sustained reads while producing no progress. It did not exit on SIGTERM.

Main process during the stall

• Alive, runtimeState: starting, listening on the configured port
• RSS flat at 6,709 MB across repeated samples (this was not a memory leak — an early reading that looked like growth was swap being paged back in)
• Last emitted trace spans were git.exec and renderer.breadcrumb, then nothing

Secondary effect worth noting

While the runtime was unreachable, a Claude agent managed by Orca began repeatedly running a recursive filesystem search for the literal string "reachable": true — apparently trying to determine why the runtime was unreachable. That search ran for 7+ minutes at ~35 MB/s. Combined with the wedged child, the host reached ~480 MB/s of page-in, 90% iowait, and 20+ processes in uninterruptible disk wait.

This is a feedback loop: the runtime being unreachable induced agent behaviour that made the host materially worse. Even with the underlying spawn bug fixed, it may be worth bounding what agents do when the runtime is unreachable.

Reproduction steps

I do not have a reliable reproduction. The spawn appears periodic/opportunistic rather than user-triggered. Observed conditions:

1. Headless orca serve on Linux, remote-runtime (pairing) mode, 3 registered worktrees.
2. Several managed terminals running Codex/Claude agents.
3. Orca spawns the rate-limit codex app-server probe.
4. Occasionally the probe does not complete; from that moment the runtime never reaches ready.

Expected behavior

A hidden rate-limit/usage subprocess should not be able to block the runtime from becoming reachable. Specifically:

• The probe should have a bounded timeout and be killed and reaped if it exceeds it.
• Failure of a usage/quota probe should degrade that feature only, not the runtime handshake.
• The runtime should reach ready regardless of probe outcome.

Actual behavior

The probe wedged and the runtime stayed at runtimeState: starting / runtimeReachable: false for 31 minutes, until the child was manually SIGKILLed. The runtime then recovered on its own with no restart, which is fairly direct evidence that the child was the blocker rather than a symptom.

Caveats / what I am not certain about

I want to be straightforward about the limits of this report:

• The host was also experiencing the every-30s repo rescan git.exec storm that was fixed in v1.4.184 (PR #14207). The box was under real I/O pressure at the time, so I cannot fully exclude that the probe wedged because of host contention rather than on its own.
• What the contention theory does not explain: the trace froze 3 seconds after this specific spawn, and killing this specific child restored the runtime with no restart.
• Since updating to v1.4.184 the git storm is gone and the probe has not been observed wedging again — but that is a short observation window (about 20 minutes at time of writing), so I cannot claim 1.4.184 fixes this.

Even if host pressure is a contributing trigger, the durable issue stands: an unbounded, unreaped usage probe can hold the runtime in starting indefinitely.

Related

• #11752 — same rate-limit-pty-cwd subsystem, different failure (proxy not propagated)
• #14207 (in v1.4.184) — the concurrent git.exec rescan storm, fixed separately

Worktree paths, hostnames, and repository names are redacted; process ids, timings, and command lines are verbatim.

Trace the runtime start path and the rate-limit-pty-cwd Codex app-server spawn, using main.trace.ndjson and the reported orca status states as entry points. Check how a stalled probe affects the runtime handshake, then verify that probe failure is bounded and the runtime can still reach ready without an Orca restart.