openai / openai/codex

app-server (Linux, remote-control): bursts of ~3,000 unreaped git child processes exhaust the user's pids.max and kill every process on the host, including Codex's own tool calls

Open
#43,256 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

app-server bug CLI performance remote
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

What version of Codex is running?

codex-cli 0.153.4 — standalone x86_64-unknown-linux-musl build, run as the headless daemon codex remote-control startcodex app-server --remote-control --listen unix://… (no Desktop app, no VS Code extension on this host).

What platform is your computer?

Linux x86_64 (kernel 7.0.0-31-generic), systemd user session; the daemon runs inside a systemd user unit whose cgroup lives under user-<uid>.slice, which carries pids.max=4096.

What issue are you seeing?

The app-server daemon periodically spawns ~3,100–3,300 git child processes in a burst and reaps them slowly, so thousands sit in state Z (<defunct>) with the daemon as parent. Each burst fills the user slice's PID budget; every process on the host — other agents' sessions, and Codex's own tool calls — then fails to fork with EAGAIN.

Measured over one boot (2026-09-05 21:03Z → 2026-09-06 10:52Z): 16 bursts, pids.events max on the slice climbing 0 → 231,939, in quanta of ≈6,400 rejected forks per burst (6,370–6,445) and small multiples — i.e. a fixed fan-out that keeps retrying at the ceiling.

Two samples taken mid-burst by a periodic cgroup snapshot (5-minute cadence):

time (UTC) tasks in the slice tasks in the daemon's cgroup by name oldest rows
05:43:04 3,273 3,197 3,124 × git all git, ppid = app-server PID, state Z (one S)
09:54:34 3,377 3,379 3,336 × git same
10:05:04 995 961 × git draining

Between samples the count falls back to ~100, so children are eventually reaped — the storm just accumulates far faster than it drains.

Effects seen from Codex's side in the same minute as the first sample:

failed to spawn code-mode host …/bin/codex-code-mode-host: Resource temporarily unavailable (os error 11)
exec_command failed: CreateProcess { message: "Rejected(\"Failed to create unified exec process: Resource temporarily unavailable (os error 11)\")" }
<script>: fork: retry: Resource temporarily unavailable

and a vitest run inside a tool call dying with EPIPE after fork retries. Three unrelated agent sessions on the same host (separate cgroups, each bounded to 512 tasks) were killed by the same EAGAIN nine seconds after the 05:43 sample.

Timing relative to turns is inconsistent: one burst began ~30 s before a user turn started (the thread had been idle for minutes), another ~30 s after one. I could not capture argv — the children live for milliseconds and the snapshot tool deliberately drops argv — so I do not know which git command this is.

Possibly relevant workspace shape, in case the fan-out is per-file like #34005: the opened repo has several large untracked directories (a worktree pool of ~370k files, plus scratch dirs), ~200 linked git worktrees and ~300 local branches. That is a hypothesis, not a measurement.

Likely related: #34005 (Desktop/Electron main thread spawning git per untracked file, <defunct> trail — same symptom shape, different component) and #25744 (unreaped zombie children on macOS).

What steps can reproduce the bug?

Not reduced to a minimal case. Observed conditions:

  1. Start the daemon: codex remote-control start on Linux, workspace = a repo with large untracked directories and many worktrees.
  2. Drive threads through remote control over several hours (multiple threads, codex-auto-review reviewer threads active).
  3. Sample pids.current of the daemon's cgroup at 1–5 s cadence, and ps -o ppid,stat,comm filtered to the daemon's PID: bursts of ~3k git children in Z appear; pids.events on the enclosing slice jumps by ≈6.4k per burst.
What is the expected behavior?
  • Children spawned by the daemon are reaped promptly (or spawned through a bounded pool), so the daemon's task count stays in the tens, not thousands.
  • The git fan-out is bounded or coalesced — one git status/diff per refresh rather than one process per file — and skips or throttles on very large untracked trees.
  • Ideally a config knob to disable or rate-limit the background git probing for the app-server.
Additional information

Host-side mitigation applied: TasksMax=2048 on the daemon's systemd unit. That confines the failure to the daemon's own cgroup (other sessions survive), but Codex still cannot complete a build while a burst is in progress, so the storm itself is the defect. Happy to run a diagnostic build or capture argv with a specific tool if that helps.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the app-server launched by codex remote-control start and codex app-server --remote-control, then reproduce while sampling pids.current, pids.events, and ps -o ppid,stat,comm for the daemon. Trace which background operation creates the git burst and verify that children are reaped promptly or that git work is bounded, without exhausting the daemon or enclosing user slice.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, linux, rust
Domain
backend, cli, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.