openai / openai/codex

codex --version on another host can delete a running server’s sandbox helpers when CODEX_HOME is on GPFS

Open
#44,361 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug CLI remote sandbox
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

What version of Codex CLI is running?

0.153.4

What subscription do you have?

Pro

Which model were you using?

astra

What platform is your computer?

Linux 4.18.0-477.27.1.el8_8.x86_64 x86_64 x86_64

What terminal emulator and version are you using (if applicable)?

None

Codex doctor report
{
  "schemaVersion": 1,
  "generatedAt": "1789001965s since unix epoch",
  "overallStatus": "warning",
  "codexVersion": "0.153.4",
  "checks": {
    "app_server.status": {
      "id": "app_server.status",
      "category": "app-server",
      "status": "warning",
      "summary": "background server socket is stale or unreachable",
      "details": {
        "app-server version": "unavailable (failed to connect to control socket)",
        "control socket": "/gpfs/home/.codex/app-server-control/app-server-control.sock",
        "daemon state dir": "/gpfs/home/.codex/app-server-daemon",
        "mode": "persistent",
        "pid file": "/gpfs/home/.codex/app-server-daemon/app-server.pid (missing)",
        "settings": "/gpfs/home/.codex/app-server-daemon/settings.json (file)",
        "status": "stale or unreachable",
        "update-loop pid file": "/gpfs/home/.codex/app-server-daemon/app-server-updater.pid (file)"
      },
      "remediation": "Run codex app-server daemon version for more details.",
      "durationMs": 3
    },
    "auth.credentials": {
      "id": "auth.credentials",
      "category": "auth",
      "status": "ok",
      "summary": "auth is configured",
      "details": {
        "auth file": "/gpfs/home/.codex/auth.json",
        "auth storage mode": "File",
        "stored API key": "false",
        "stored ChatGPT tokens": "true",
        "stored agent identity": "false",
        "stored auth mode": "chatgpt"
      },
      "remediation": null,
      "durationMs": 0
    },
    "config.load": {
      "id": "config.load",
      "category": "config",
      "status": "ok",
      "summary": "config loaded",
      "details": {
        "CODEX_HOME": "/gpfs/home/.codex",
        "config.toml": "/gpfs/home/.codex/config.toml",
        "config.toml parse": "ok",
        "cwd": "/gpfs/home",
        "enabled feature flags": "shell_tool, view_image, sleep_tool, unified_exec, unified_exec_zsh_fork, shell_snapshot, content_item_kinds, code_mode_host, terminal_resize_reflow, sqlite, hooks, enable_request_compression, unbounded_connection_retries, multi_agent, apps, tool_search_always_defer_mcp_tools, tool_suggest, plugins, in_app_browser, in_app_chat, in_app_dictation, in_app_local_automation, in_app_updates, browser_use, browser_use_full_cdp_access, browser_use_external, computer_use, remote_plugin, plugin_sharing, image_generation, resize_all_images, item_ids, skill_mcp_dependency_install, skill_search, mentions_v2, steer, guardian_approval, goals, collaboration_modes, tool_call_mcp_elicitation, auth_elicitation, personality, fast_mode, tui_app_server, remote_compaction_v2, compaction_image_budget, workspace_dependencies",
        "feature flag overrides": "none",
        "feature flags enabled": "47",
        "log dir": "/gpfs/home/.codex/log",
        "mcp servers": "0",
        "model": "gpt-6-astra",
        "model provider": "openai",
        "sqlite home": "/gpfs/home/.codex"
      },
      "remediation": null,
      "durationMs": 0
    },
    "git.environment": {
      "id": "git.environment",
      "category": "git",
      "status": "ok",
      "summary": "git version 2.16.0",
      "details": {
        "PATH git #1": "/gpfs/share/apps/git/2.16.0/bin/git",
        "PATH git #2": "/gpfs/share/apps/git/2.49.0/bin/git",
        "PATH git #3": "/usr/bin/git",
        "PATH git entries": "3",
        "git build options": "git version 2.16.0; cpu: x86_64; no commit associated with this build; sizeof-long: 8",
        "git exec path": "/gpfs/share/apps/git/2.17.0/libexec/git-core",
        "git version": "git version 2.16.0",
        "repo detected": "false",
        "selected git": "/gpfs/share/apps/git/2.16.0/bin/git"
      },
      "remediation": null,
      "durationMs": 276
    },
    "installation": {
      "id": "installation",
      "category": "install",
      "status": "ok",
      "summary": "installation looks consistent",
      "details": {
        "PATH codex #1": "/gpfs/home/.local/bin/codex",
        "PATH codex #2": "/gpfs/home/.local/bin/codex",
        "PATH codex entries": "2",
        "current executable": "/gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl/bin/codex",
        "install context": "standalone (unix, package /gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl, bin /gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl/bin, resources /gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl/codex-resources, path /gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl/codex-path)",
        "managed by Vite+": "false",
        "managed by bun": "false",
        "managed by npm": "false",
        "managed by pnpm": "false",
        "managed package root": "not set"
      },
      "remediation": null,
      "durationMs": 15
    },
    "mcp.config": {
      "id": "mcp.config",
      "category": "mcp",
      "status": "ok",
      "summary": "no MCP servers configured",
      "details": {},
      "remediation": null,
      "durationMs": 0
    },
    "network.env": {
      "id": "network.env",
      "category": "network",
      "status": "ok",
      "summary": "network-related environment looks readable",
      "details": {
        "managed proxy": "not configured",
        "proxy env vars": "none",
        "respect system proxy": "disabled"
      },
      "remediation": null,
      "durationMs": 0
    },
    "network.provider_reachability": {
      "id": "network.provider_reachability",
      "category": "reachability",
      "status": "ok",
      "summary": "active provider endpoints are reachable over HTTP",
      "details": {
        "ChatGPT inference URL": "https://chatgpt.com/backend-api/<redacted> reachable (HTTP 405)",
        "reachability mode": "ChatGPT auth"
      },
      "remediation": null,
      "durationMs": 368
    },
    "network.websocket_reachability": {
      "id": "network.websocket_reachability",
      "category": "websocket",
      "status": "ok",
      "summary": "Responses WebSocket handshake succeeded",
      "details": {
        "DNS": "2 IPv4, 2 IPv6, first IPv4",
        "auth mode": "chatgpt",
        "connect timeout": "15000 ms",
        "endpoint": "wss://chatgpt.com/backend-api/<redacted>",
        "handshake result": "HTTP 101 Switching Protocols",
        "model provider": "openai",
        "provider name": "OpenAI",
        "proxy env vars": "none",
        "reasoning header": "false",
        "server model present": "false",
        "supports websockets": "true",
        "wire API": "responses"
      },
      "remediation": null,
      "durationMs": 974
    },
    "runtime.provenance": {
      "id": "runtime.provenance",
      "category": "runtime",
      "status": "ok",
      "summary": "running standalone on linux-x86_64",
      "details": {
        "commit": "unknown",
        "current executable": "/gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl/bin/codex",
        "install method": "standalone (unix, package /gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl, bin /gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl/bin, resources /gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl/codex-resources, path /gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl/codex-path)",
        "platform": "linux-x86_64",
        "version": "0.153.4"
      },
      "remediation": null,
      "durationMs": 0
    },
    "runtime.search": {
      "id": "runtime.search",
      "category": "search",
      "status": "ok",
      "summary": "search is OK (bundled)",
      "details": {
        "search command": "/gpfs/home/.codex/packages/standalone/releases/0.153.4-x86_64-unknown-linux-musl/codex-path/rg",
        "search command readiness": "file exists",
        "search provider": "bundled"
      },
      "remediation": null,
      "durationMs": 0
    },
    "sandbox.helpers": {
      "id": "sandbox.helpers",
      "category": "sandbox",
      "status": "ok",
      "summary": "sandbox configuration is readable",
      "details": {
        "approval policy": "OnRequest",
        "codex-linux-sandbox helper": "/gpfs/home/.codex/tmp/arg0/codex-arg0AlNYXZ/codex-linux-sandbox",
        "execve wrapper helper": "/gpfs/home/.codex/tmp/arg0/codex-arg0AlNYXZ/codex-execve-wrapper",
        "filesystem sandbox": "restricted",
        "network sandbox": "enabled"
      },
      "remediation": null,
      "durationMs": 0
    },
    "security.endpoint": {
      "id": "security.endpoint",
      "category": "security",
      "status": "ok",
      "summary": "endpoint protection is not inspected on this platform",
      "details": {
        "endpoint products": "not inspected on this platform"
      },
      "remediation": null,
      "durationMs": 0
    },
    "state.paths": {
      "id": "state.paths",
      "category": "state",
      "status": "ok",
      "summary": "state paths and databases are inspectable",
      "details": {
        "CODEX_HOME": "/gpfs/home/.codex (dir)",
        "active rollout files": "561 files, 4553817394 total bytes, 8117321 average bytes",
        "archived rollout files": "0 files, 0 total bytes, 0 average bytes",
        "goals DB": "/gpfs/home/.codex/goals_1.sqlite (file)",
        "goals DB integrity": "ok",
        "log DB": "/gpfs/home/.codex/logs_2.sqlite (file)",
        "log DB integrity": "ok",
        "log dir": "/gpfs/home/.codex/log (dir)",
        "memories DB": "/gpfs/home/.codex/memories_1.sqlite (file)",
        "memories DB integrity": "ok",
        "queue DB": "/gpfs/home/.codex/queue_1.sqlite (file)",
        "queue DB integrity": "ok",
        "sqlite home": "/gpfs/home/.codex (dir)",
        "standalone release cache": "6 entries in /gpfs/home/.codex/packages/standalone/releases",
        "state DB": "/gpfs/home/.codex/state_5.sqlite (file)",
        "state DB integrity": "ok",
        "thread history DB": "/gpfs/home/.codex/thread_history_1.sqlite (file)",
        "thread history DB integrity": "ok"
      },
      "remediation": null,
      "durationMs": 20288
    },
    "state.rollout_db_parity": {
      "id": "state.rollout_db_parity",
      "category": "threads",
      "status": "ok",
      "summary": "rollout files and state DB thread inventory agree",
      "details": {
        "default model provider": "openai",
        "rollout DB active files": "561",
        "rollout DB active rows": "561",
        "rollout DB archive mismatches": "0",
        "rollout DB archived files": "0",
        "rollout DB archived rows": "0",
        "rollout DB duplicate DB paths": "0",
        "rollout DB duplicate rollout thread ids": "0",
        "rollout DB malformed file names": "0",
        "rollout DB missing active rows": "0",
        "rollout DB missing archived rows": "0",
        "rollout DB model providers": "openai=561",
        "rollout DB rows": "561",
        "rollout DB scan cap reached": "false",
        "rollout DB scan errors": "0",
        "rollout DB sources": "subagent:thread_spawn=511, vscode=40, cli=10",
        "rollout DB stale rows": "0"
      },
      "remediation": null,
      "durationMs": 15102
    },
    "system.disk": {
      "id": "system.disk",
      "category": "disk",
      "status": "ok",
      "summary": "sufficient free disk space (11093727.7 GiB)",
      "details": {
        "CODEX_HOME available": "11093727.7 GiB",
        "failure threshold": "1.0 GiB",
        "warning threshold": "5.0 GiB",
        "worktree available": "11093727.7 GiB"
      },
      "remediation": null,
      "durationMs": 0
    },
    "system.environment": {
      "id": "system.environment",
      "category": "system",
      "status": "ok",
      "summary": "OS language en-US",
      "details": {
        "EDITOR": "not set",
        "LANG": "en_US.UTF-8",
        "VISUAL": "not set",
        "os": "Red Hat Enterprise Linux 8.8.0 (Ootpa) [64-bit]",
        "os language": "en-US",
        "os type": "Red Hat Enterprise Linux",
        "os version": "8.8.0"
      },
      "remediation": null,
      "durationMs": 125
    },
    "terminal.env": {
      "id": "terminal.env",
      "category": "terminal",
      "status": "ok",
      "summary": "terminal metadata was detected",
      "details": {
        "SSH_CLIENT": "present",
        "SSH_CONNECTION": "present",
        "SSH_TTY": "present",
        "TERM": "xterm-256color",
        "color output": "enabled",
        "effective locale": "en_US.UTF-8",
        "stderr is terminal": "true",
        "stdin is terminal": "true",
        "stdout is terminal": "true",
        "terminal": "unknown",
        "terminal size": "100x32"
      },
      "remediation": null,
      "durationMs": 0
    },
    "terminal.title": {
      "id": "terminal.title",
      "category": "title",
      "status": "ok",
      "summary": "terminal title default",
      "details": {
        "terminal title activity": "true",
        "terminal title items": "activity, project-name",
        "terminal title project source": "cwd",
        "terminal title project value": "username",
        "terminal title source": "default"
      },
      "remediation": null,
      "durationMs": 0
    },
    "updates.status": {
      "id": "updates.status",
      "category": "updates",
      "status": "ok",
      "summary": "update configuration is locally consistent",
      "details": {
        "cached latest version": "0.153.4",
        "check for update on startup": "true",
        "last checked at": "2026-09-06T01:54:05.551042253Z",
        "latest version": "0.154.0",
        "latest version status": "newer version is available",
        "update action": "standalone installer",
        "version cache": "/gpfs/home/.codex/version.json"
      },
      "remediation": null,
      "durationMs": 264
    }
  }
}
What issue are you seeing?

With Codex CLI 0.153.4 on Linux, running “codex --version” on one host can delete temporary helper files belonging to a live Codex app-server on another host when both use the same CODEX_HOME on a shared GPFS filesystem.

The app-server remains running, but subsequent sandboxed commands fail until the server is restarted.

This occurred while using Codex Desktop over SSH.

ENVIRONMENT

Codex CLI: 0.153.4
Binary distribution: x86_64-unknown-linux-musl
Remote OS: Linux
Shared filesystem: GPFS / IBM Storage Scale
Two hosts accessing the same home directory and default CODEX_HOME
Persistent Codex app-server on host A
Short-lived “codex --version” invocation on host B

The original environment used a scheduler-managed compute allocation. That allocation was still running when the failure occurred.

OBSERVED FAILURE

Ordinary exec_command calls failed before the requested shell command started:

Failed to create unified exec process: No such file or directory (os error 2)

A TTY-enabled attempt exposed the missing executable:

Unable to spawn <CODEX_HOME>/tmp/arg0/codex-arg0/codex-linux-sandbox because it doesn't exist on the filesystem

Inspection of the live app-server showed:

/proc//fd/3 -> <CODEX_HOME>/tmp/arg0/codex-arg0/.lock (deleted)

Its fdinfo still showed an advisory FLOCK write lock. Commands executed successfully outside the sandbox.

CONTROLLED REPRODUCTION RESULTS

A disposable CODEX_HOME was used to avoid touching the real runtime.

While host A held the helper directory’s .lock file:

Running “codex --version” on host A preserved the directory.
Running the same Codex version on host B deleted the directory.

A separate lock test showed:

Another process on host A was blocked from acquiring the lock.
A process on host B successfully acquired the lock while host A still held it.

This reproduces the deletion mechanism.

What steps can reproduce the bug?

MINIMAL REPRODUCTION

Prerequisites:

Two Linux hosts with a shared home directory on the affected filesystem.
Codex 0.153.4 available as “codex” on both hosts, including noninteractive SSH.
Working SSH from host A to host B.
Both hosts resolve the temporary directory below to the same shared files.

Save the following Python script as reproduce.py on host A, then run:

python3 reproduce.py host-b

The script uses an isolated temporary CODEX_HOME. It creates a synthetic helper directory with a held lock; no live Codex server is needed.

BEGIN PYTHON SCRIPT

import fcntl
import os
from pathlib import Path
import shlex
import subprocess
import sys
import tempfile

host_b = sys.argv[1]

Must be on storage shared by both hosts.

Use the home directory because Codex rejects CODEX_HOME under /tmp.

with tempfile.TemporaryDirectory(
prefix="codex-helper-repro-", dir=Path.home()
) as temporary_home:
codex_home = Path(temporary_home)
helper = codex_home / "tmp" / "arg0" / "codex-arg0-repro"
helper.mkdir(parents=True)
marker = helper / "marker"
marker.write_text("Disposable reproduction directory\n")

with (helper / ".lock").open("w+") as lock:
    fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)

    env = os.environ.copy()
    env["CODEX_HOME"] = str(codex_home)

    print("Version check on host A:", flush=True)
    subprocess.run(["codex", "--version"], env=env, check=True)
    print("Helper exists after host A:", helper.exists(), flush=True)

    if not marker.exists():
        raise RuntimeError("Same-host control already removed the helper")

    # Confirm host B can see the marker before invoking Codex there.
    remote_command = (
        "test -f " + shlex.quote(str(marker))
        + " && "
        + shlex.join([
            "env",
            "CODEX_HOME=" + str(codex_home),
            "codex",
            "--version",
        ])
    )

    print("Version check on host B:", flush=True)
    subprocess.run(
        [
            "ssh", "-T",
            "-o", "BatchMode=yes",
            "-o", "ConnectTimeout=10",
            host_b,
            remote_command,
        ],
        check=True,
        timeout=60,
    )

    print("Helper exists after host B:", helper.exists())
    print("Marker exists after host B:", marker.exists())

END PYTHON SCRIPT

Expected output:

Helper exists after host A: True
Helper exists after host B: True
Marker exists after host B: True

Observed on the affected filesystem:

Helper exists after host A: True
Helper exists after host B: False
Marker exists after host B: False

The compute-side process holds its lock throughout both invocations.

What is the expected behavior?

SOURCE-LEVEL EXPLANATION

Source inspected: openai/codex, tag rust-v0.153.4.

Relevant files:

codex-rs/cli/src/main.rs
codex-rs/arg0/src/lib.rs

In this version, helper initialization happens before CLI argument parsing, so even “--version” runs cleanup.

The janitor_cleanup function examines helper directories and attempts to acquire each directory’s .lock file. If it acquires the lock, it recursively deletes that directory.

On the tested GPFS deployment, that lock did not exclude a process on another host. Cleanup therefore treated a live server’s directory as unused.

This does not establish whether the cross-node lock behavior is expected GPFS behavior or deployment-specific. It establishes that the cleanup protection is insufficient in this environment.

EXPECTED BEHAVIOR AND POSSIBLE FIXES

Running “codex --version” should not remove another live process’s helpers.

Cleanup should handle shared CODEX_HOME directories safely, potentially by scoping helpers and cleanup by host.

Version and help commands could bypass helper initialization.

A running server should recover or provide actionable diagnostics when its cached helper path disappears.

Additional information

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with CODEX_HOME on GPFS by running codex --version from another host while a server is active. Start with the app-server and sandbox-helper lifecycle paths, using the reported helper files under CODEX_HOME/tmp; done means the command no longer removes helpers needed by the running server.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
operating-systems, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.