drop-sudo (default safety-strategy) chmods /run service sockets to 0700, breaking D-Bus and systemd-resolved — GitHub-hosted runners die mid-job at ~55 minutes

Summary

The default safety-strategy: drop-sudo restricts every root-owned service socket under /run that the runner user can write to (src/dropSudo.ts, restrictRootServiceSocket: fs.chmod(fd, stats.mode & 0o700)). On Ubuntu runners this removes group/other access from /run/dbus/system_bus_socket and /run/systemd/journal/stdout, among others.

The consequence is that systemd-resolved — which runs as the non-root systemd-resolve user — can no longer connect to the system bus. The already-running instance keeps serving until anything restarts it; from that moment it crash-loops and hits the systemd start limit, and name resolution on the machine is permanently dead.

On GitHub-hosted runners this manifests as jobs failing with:

The job was not started because... / The hosted runner lost communication with the server. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

The runner agent survives on its established connections for a while, then dies — in our data, consistently 52–65 minutes into the job. This reproduces on both standard ubuntu-latest and 4-core/16GB larger runners.

The action's own documentation hints at host mutation ("account and host changes can outlive a job on reused self-hosted runners... access to existing root-owned service sockets under /run... is also removed"), but nothing warns that it degrades the hosted runner it is currently executing on, or that the runner agent itself is a casualty.

Evidence

Statistical (GitHub-hosted, one org, 4-day window):

• 12 of 12 jobs containing a codex-action step that ran past 45 minutes died with "lost communication", deaths spread 52–64.5 min. Standard and larger runners interleaved.
• 94 of 94 jobs under 5 minutes succeeded (including many that ran the same action briefly).
• Control: a job doing nothing but a shell heartbeat survived 75 minutes on both runner types. So it is not a duration limit — it requires the workload.
• A sibling workflow in the same org that invokes the Codex CLI directly (no codex-action, hence no drop-sudo) has never exhibited the failure, including jobs with a 240-minute budget.

Direct observation (self-hosted Ubuntu 24.04.4 VM, runner agent 2.336.0, codex-action@v1, Codex CLI 0.149.1):

Minutes after a job ran the codex-action step with drop-sudo, systemd-resolved restarted and crash-looped:

(resolved)[5596]: systemd-resolved.service: Failed to connect stdout to the journal socket, ignoring: Permission denied
systemd[1]: systemd-resolved.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: systemd-resolved.service: Start request repeated too quickly.
systemd[1]: Failed to start systemd-resolved.service - Network Name Resolution.

Running the binary by hand as root shows the fatal error:

Failed to connect to system bus: Permission denied
Could not create manager: Permission denied

getent hosts github.com fails for every process from that point; querying the cloud metadata resolver directly still works, i.e. only the local resolver stack is destroyed. Two identically-provisioned VMs died the same way within one second of each other (a timed service restart hit the pre-poisoned state on both).

Second reuse hazard on the same machine: removeUserFromSudoersD() strips the runner user's entries from /etc/sudoers.d/*. On the next job, the action's own setup fails at its sudo chmod 444 "$SERVER_INFO_FILE" step:

sudo: a terminal is required to read the password; either use the -S option ...
sudo: a password is required

So on a reused runner the action breaks its own subsequent invocations, not just the host.

Minor related reuse bug: the action appends its proxy block to ~/.codex/config.toml on every run; a second run on the same machine fails with config.toml:6:1: duplicate key — model_provider.

Why the hosted-runner death is delayed ~50 minutes (inference)

The runner agent's long-poll survives on established connections after DNS dies. The 52–65-minute death band is consistent with the agent's periodic session/credential refresh requiring functional name resolution and failing. We don't have agent-side logs to confirm the exact mechanism — happy to share run IDs privately with maintainers or GitHub so it can be confirmed from telemetry.

Minimal repro (any Ubuntu 24.04 VM or hosted runner)

1. Run a workflow step with uses: openai/codex-action@v1, defaults (drop-sudo), any trivial prompt.
2. Afterwards: ls -l /run/dbus/system_bus_socket → group/other bits stripped.
3. sudo systemctl restart systemd-resolved → crash loop, Failed to connect to system bus: Permission denied.
4. getent hosts github.com → fails machine-wide.
5. On a hosted runner, instead let the job continue >50 minutes after the codex step: the job dies with "lost communication".

Suggested fixes / mitigations

1. Scope the restriction. stats.mode & 0o700 removes access for every non-root user, including system service peers (messagebus, systemd-resolve). Restricting only the specific group(s) the runner user belongs to — or removing the runner user from those groups without touching socket modes — would achieve the stated goal without destroying system IPC.
2. Don't mutate host state at all: confining Codex in a namespace/sandbox (the action already ships bwrap for workspace-write) achieves isolation without irreversible host changes.
3. At minimum, document loudly that drop-sudo degrades the current host, that long-running jobs on GitHub-hosted runners will be declared lost, and that reused self-hosted runners will fail subsequent invocations (sudoers stripping, config.toml duplicate key).
4. Consider failing fast or warning when --ephemeral/fresh-VM cannot be assumed.

Workaround we ended up with

• safety-strategy set away from drop-sudo (relying on the workspace-write bwrap sandbox as the containment boundary), via a repository variable so it's tunable per environment.
• On self-hosted runners: a static /etc/resolv.conf pointing at the cloud metadata resolver (removes systemd-resolved from the datapath entirely), plus a pre-job hook wiping ~/.codex.

Versions: codex-action@v1 (current), Codex CLI 0.149.1, runner agent 2.336.0, Ubuntu 24.04 (hosted image 20260816.277.1 and stock cloud image).

Start in src/dropSudo.ts at restrictRootServiceSocket and inspect how fs.chmod applies 0700 to root-owned sockets under /run. Reproduce on Ubuntu with the listed socket, systemd-resolved restart, and getent hosts checks, then verify the chosen change no longer breaks system IPC or reused-runner behavior while preserving the intended safety restriction.