codex-companion: per-job state JSON stuck at status=running after task_complete; streaming .output drain disconnects mid-turn

Summary

A codex-companion task invocation can have its streaming .output file silently stop updating mid-turn while Codex itself keeps running. When the task later reaches task_complete successfully, the per-job state JSON at ~/.claude/plugins/data/codex-inline/state/<workspace>/jobs/task-<id>.json is never updated from "status": "running" to a terminal state. The PID in the JSON remains pointed at a now-dead process. The task's findings are fully recoverable from the canonical rollout JSONL at ~/.codex/sessions/YYYY/MM/DD/rollout-<timestamp>-<turn_id>.jsonl (task_complete event's last_agent_message field), but the bridge-layer signal that the task finished is lost.

Observed incident

• 2026-04-24 on macOS 15.0.2 / Darwin 25.3.0; plugin v1.0.2 (gitCommitSha 8e403f9d4b496b0b0aff50fa3673f889f6a22cb1); codex-cli v0.118.0.

• Codex-companion job id: task-mocfoox2-12xdjb. Codex thread id: 019dbdd4-b9bf-7af3-9275-610b76fd1ec6.

• Streaming .output at /private/tmp/claude-<uid>/.../tasks/bkt2c8mcy.output stopped growing at 04:54:03Z after 72 lines; final entry was a normal Command completed: ... (exit 0). No error marker, no EOF indicator, no termination log line.

• Per-job log at ~/.claude/plugins/data/codex-inline/state/<workspace>/jobs/task-mocfoox2-12xdjb.log also stopped at 04:54:03Z.

• Codex continued running for ~11 more minutes. Canonical rollout JSONL at ~/.codex/sessions/2026/04/24/rollout-2026-04-24T07-52-08-019dbdd4-b9bf-7af3-9275-610b76fd1ec6.jsonl has the task_complete event at 05:05:07.319Z with the full last_agent_message payload (9,888 chars, a substantive review output).

• At time of incident-observation (~20 min after drain stopped), the companion state JSON still reads:

  {
    "status": "running",
    "phase": "running",
    "pid": 41501,
    "threadId": "019dbdd4-b9bf-7af3-9275-610b76fd1ec6",
    "turnId": "019dbdd4-ba6c-7282-9a51-4e1e55fddecc",
    ...
  }
  

  kill -0 41501 → 1 (no such process). PID is stale.

• Broker log at /var/folders/.../cxc-X56GUo/broker.log is 0 bytes — no diagnostic trail at the bridge layer either.

Consequences

1. Caller cannot distinguish stalled vs completed tasks via companion-visible state. Recovery requires reaching for the canonical rollout JSONL, which isn't documented as a fallback.
2. codex-companion status --all --json reports dead tasks as running indefinitely, polluting any "what's currently running" view. In our incident we spawned a second diagnosis task expecting to find a stalled job; the status API reported no running jobs (so the entry had already been cleared from status --all --json's view somewhere in companion logic) yet the per-job JSON file still showed running. So the inconsistency is worse: status --all --json and the on-disk JSON disagree with each other on the same job.
3. Cancel helpers misbehave on ghost tasks (observed adjacent symptom: codex-companion cancel <task> failed with an environment error on the stuck job — possibly a separate bug, but the state divergence is the root enabler).

Suggested fixes (any of these closes the gap)

• (a) Broker-side terminal marker on pipe close. When the broker detects the Codex-CLI-side pipe has closed, write a terminal event (e.g., type=task_ended with reason=pipe_closed) to both the streaming .output AND the per-job state JSON before shutting down the job slot. This would also surface unexpected crashes, which today produce zero on-disk evidence (broker.log is 0 bytes in this incident).
• (b) Companion-side liveness reconciler. Before codex-companion status returns, walk jobs/*.json, kill -0 pid for any status=running entries, and if the PID is gone, try to find the matching rollout JSONL (by threadId / turnId) and either (i) update to a terminal state from the rollout's task_complete, or (ii) mark status=unknown (pid_gone_no_rollout_match) so callers can at least tell it apart from an actively-running task.
• (c) Documented recovery path. Independent of any fix, document the rollout JSONL as the canonical transcript in the companion's recovery docs — callers hitting this today have no reliable signal that a "still running" job is actually done-and-stuck-in-state, and no documented path to find the transcript.

Reproduction

I don't have a clean deterministic reproducer. The incident happened on a ~5-minute review task that did a lot of file reads and eventually produced a long last_agent_message. The drain stopped mid-task (after ~1.5 min of active tool-use). Candidate triggers: broker-side backpressure, Codex CLI burst that temporarily saturated the pipe, or an unrelated broker hiccup. Will update if reproducible.

Workaround

Recover from canonical rollout JSONL using threadId / turnId from the stuck companion state JSON:

import json

rollout_path = "~/.codex/sessions/YYYY/MM/DD/rollout-<timestamp>-<turn_id>.jsonl"
with open(rollout_path) as f:
    for line in f:
        evt = json.loads(line)
        if evt.get("type") == "event_msg" and evt["payload"].get("type") == "task_complete":
            print(evt["payload"]["last_agent_message"])

The rollout filename embeds <timestamp>-<turn_id>; matches the turnId field in the stuck companion state JSON. The threadId field in the JSON also matches the session id at the top of the rollout file (type=session_meta event's payload.id).

Start by tracing the broker pipe-close path and codex-companion status --all --json handling; the issue names no source files. Reproduce or inspect the stuck job state JSON alongside the canonical rollout JSONL under ~/.codex/sessions/. Done means completed or dead jobs no longer remain indistinguishable as running, and the state and status views agree.