Codex execution context cannot access host NVIDIA GPUs after launch or resume

What version of the Codex App are you using (From “About Codex” dialog)?

codex-cli 0.151.0-alpha.7.2

What subscription do you have?

pro

What platform is your computer?

No response

What issue are you seeing?

Summary

Codex is launched from a Linux shell where NVIDIA GPUs are available, but commands executed inside the affected Codex session cannot access the same GPU devices.

The GPU task therefore cannot continue and reports:

WAITING_OUTER_GPU_RUNNER

WAITING_OUTER_GPU_RUNNER is a status emitted by my task's fail-closed GPU preflight workflow. I am including it as an observable symptom; I am not claiming that it is an official Codex status code.

The underlying problem is that the parent shell can see the NVIDIA GPUs, while the execution context used by Codex cannot see /dev/nvidia* or any CUDA devices.

Environment

• Component: Codex CLI
• Launch mode: codex or codex resume
• Operating system: Linux
• Runtime environment: Kubernetes/container-based GPU server
• GPUs: 4 × NVIDIA A100-SXM4-80GB
• Codex version: codex-cli 0.151.0-alpha.7.2
• Python version: Python 3.12.3
• PyTorch version: 2.8.0+cu128
• CUDA runtime reported by PyTorch: 12.8
• Authentication: ChatGPT Pro account with an active paid subscription

What steps can reproduce the bug?

Steps to reproduce

1. Verify GPU access in the parent shell

Before launching Codex, run:

nvidia-smi -L

python - <<'PY'
import torch

print("torch_version:", torch.__version__)
print("torch_cuda_version:", torch.version.cuda)
print("cuda_available:", torch.cuda.is_available())
print("cuda_device_count:", torch.cuda.device_count())
PY

The parent shell reports four NVIDIA A100 GPUs:

GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB

torch_version: 2.8.0+cu128
torch_cuda_version: 12.8
cuda_available: True
cuda_device_count: 4

2. Launch Codex from the same shell

Start a new Codex session:

codex

The problem can also occur when resuming an existing session:

codex resume

3. Run the same GPU preflight from the Codex task

Ask Codex to run:

ls -l /dev/nvidia* 2>&1
nvidia-smi -L

python - <<'PY'
import os
import torch

print("CUDA_VISIBLE_DEVICES:", os.environ.get("CUDA_VISIBLE_DEVICES"))
print("NVIDIA_VISIBLE_DEVICES:", os.environ.get("NVIDIA_VISIBLE_DEVICES"))
print(
    "CODEX_INNER_DEVICES_MASKED:",
    os.environ.get("CODEX_INNER_DEVICES_MASKED"),
)
print("torch_version:", torch.__version__)
print("torch_cuda_version:", torch.version.cuda)
print("cuda_available:", torch.cuda.is_available())
print("cuda_device_count:", torch.cuda.device_count())
PY

Actual behavior

In affected Codex sessions, one or more of the following occurs:

/dev/nvidia* is missing
nvidia-smi cannot communicate with the NVIDIA driver
torch.cuda.is_available() == False
torch.cuda.device_count() == 0

This happens even though the parent shell from which Codex was launched can access all four GPUs.

The model code, Python environment, CUDA installation, and physical server have not changed between the parent-shell check and the Codex-executed check. The observed difference is the process, container, or execution context in which the command runs.

The task then stops safely with:

FINAL_STATUS=WAITING_OUTER_GPU_RUNNER

GPU correctness and performance tests cannot legitimately be replaced with CPU execution or synthetic results.

What is the expected behavior?

Expected behavior

When Codex is launched from a GPU-enabled shell, commands executed by the Codex session should inherit or retain access to the GPU device nodes and relevant CUDA environment.

At minimum, Codex should:

• Preserve /dev/nvidia* device visibility when creating its command-execution context.
• Preserve the applicable CUDA_VISIBLE_DEVICES and NVIDIA_VISIBLE_DEVICES settings.
• Clearly report when GPU devices are intentionally masked or cannot be inherited.
• Explain the meaning and intended use of CODEX_INNER_DEVICES_MASKED, if it is set by Codex.
• Provide a supported way to restart the command runner inside the same conversation without losing task context.
• Detect when a resumed session is attached to a different container or device namespace from the shell that launches codex resume.

Additional information

Additional diagnostics

The following comparison may help identify whether Codex is switching mount, PID, cgroup, or container namespaces:

echo "=== PROCESS ==="
id
pwd

echo "=== CGROUP ==="
cat /proc/self/cgroup

echo "=== NAMESPACES ==="
readlink /proc/self/ns/mnt
readlink /proc/self/ns/pid
readlink /proc/self/ns/cgroup

echo "=== NVIDIA DEVICES ==="
ls -l /dev/nvidia* 2>&1

echo "=== NVIDIA SMI ==="
nvidia-smi -L 2>&1

echo "=== RELEVANT ENVIRONMENT ==="
env | rg \
  '^(CUDA_VISIBLE_DEVICES|NVIDIA_VISIBLE_DEVICES|CODEX_INNER_DEVICES_MASKED)=' \
  || true

I can provide sanitized output from both the parent shell and the affected Codex execution context. Internal hostnames, repository paths, credentials, and tokens will be redacted.

Frequency and impact

This has occurred repeatedly across GPU-dependent Codex tasks.

It blocks real validation for vLLM, SGLang, Megatron-LM, and Agentic RL workloads. These tasks require real CUDA execution for correctness, distributed behavior, memory measurement, and performance benchmarking.

Changing the Python environment or reinstalling CUDA does not address the problem when the affected execution context has no NVIDIA device nodes.

Starting a completely new Codex process from a shell that can see the GPUs has restored GPU access in at least one previous occurrence. This suggests that the problem is related to runner or execution-context inheritance rather than a broken CUDA or PyTorch installation.

Possibly related issue

A separate problem sometimes interrupts running tasks with:

Selected model is at capacity. Please try a different model.

That model-capacity and task-recovery problem is tracked separately in:

• https://github.com/openai/codex/issues/41810

The two problems may occur during the same workflow, especially after restarting or resuming a task, but I currently have no evidence that they share the same root cause.

Questions

1. Does Codex create an inner command runner or container namespace that can mask /dev/nvidia*?
2. Is CODEX_INNER_DEVICES_MASKED set by Codex, and what behavior is it intended to control?
3. Can codex resume reconnect a conversation to a different execution context from the shell that invoked it?
4. What is the supported method for preserving GPU device access across a resumed Codex session?

Start at the codex and codex resume entry points and compare the parent-shell diagnostics with the Codex task diagnostics for namespaces, /dev/nvidia*, and the relevant environment variables. Reproduce with nvidia-smi and the PyTorch CUDA checks in a new and resumed session; done means the execution context preserves GPU access or clearly reports why devices are masked.