openai / openai/codex

Remote SSH reports `socket hang up` when remote app-server cannot initialize SQLite state on NFS CODEX_HOME

Open
#44,950 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

app app-server bug connectivity remote
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

What issue are you seeing?

The remote daemon fails during SQLite state initialization on NFS.

Desktop's bootstrap reports success, but the control socket is absent. codex app-server proxy then exits with ENOENT, and Desktop surfaces the result as a generic HTTP-client ECONNRESET / socket hang up.

In more detail
Codex Desktop Remote SSH successfully authenticates to the Linux host, but the connection eventually fails with:

socket hang up

Normal OpenSSH connectivity is healthy. For example:

ssh h2codex-test

successfully opens a shell on the remote compute node.

These also succeed:

ssh h2codex-test "codex --version"
codex-cli 0.154.0

and:

ssh h2codex-test "codex app-server --help"

prints the expected app-server help.

However, the Desktop Remote SSH flow does more than those simple probes. It bootstraps the remote Codex daemon and then runs:

codex app-server proxy

over SSH.

The proxy failed because the expected control socket did not exist:

Error: failed to connect to socket at
/u/home/a/adtolton/.codex/app-server-control/app-server-control.sock

Caused by:
No such file or directory (os error 2)

The remote app-server log showed the underlying daemon startup failure:

Error: failed to initialize sqlite state runtime under
/u/home/a/adtolton/.codex:
failed to initialize state runtime at
/u/home/a/adtolton/.codex

/u/home/a/adtolton is NFS-backed.

Desktop logged bootstrap exit code 0, then the proxy exited because the socket was absent. The Desktop UI surfaced this downstream failure only as:

socket hang up

What steps can reproduce the bug?
What version of Codex CLI is running?

codex-cli 0.154.0

What platform is the client?

Windows 11

Codex Desktop / ChatGPT desktop:
26.903.9818.0

Windows OpenSSH:
OpenSSH_for_Windows_9.5p2

What platform is the remote host?

Linux 5.14.0-687.38.1.el9_8.x86_64

The remote home directory is mounted over NFSv3.

The remote session was running on a Slurm compute node. Virtual memory on the compute node was not constrained, so this is separate from the Code Mode / RLIMIT_AS issue reported in #43820.

Steps to reproduce

Use a remote Linux host whose home directory / CODEX_HOME is on NFS.
Install Codex CLI 0.154.0 on the remote host.

Verify that simple SSH and CLI probes work:

codex --version
codex app-server --help
Connect to that host through Codex Desktop Remote SSH.
Allow Desktop to bootstrap the remote app-server.

In my environment, the daemon fails to initialize SQLite state under the NFS-backed CODEX_HOME.

The app-server control socket is never created, the proxy exits, and Desktop reports:

socket hang up

What is the expected behavior?

At minimum, one of the following:

Codex app-server should support SQLite state on common NFS-backed home directories; or
Codex should detect that the state directory is unsuitable and provide a supported way to place mutable runtime/database state on node-local storage while retaining configuration elsewhere.

Additionally, Desktop should not reduce this failure to only:

socket hang up

If remote daemon startup fails, the actual initialization failure should be surfaced.

The bootstrap path should also verify that the remote daemon is actually ready and that the app-server control socket exists before reporting bootstrap success.

Additional information

Moving CODEX_HOME to node-local /tmp immediately restored the Remote SSH connection.

This appears to be two issues:

Codex mutable SQLite state is not robust to this NFS-backed home environment.
Desktop Remote SSH does not propagate the daemon startup failure and instead presents a misleading socket hang up.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce on a Linux host with an NFSv3 home backing CODEX_HOME, using the documented codex --version, codex app-server --help, and codex app-server proxy probes. Start with SQLite state initialization and the Remote SSH bootstrap readiness path; done means the supported state-location behavior is defined and the actual daemon failure is surfaced instead of only reporting socket hang up.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust, sqlite
Domain
backend-api-design, databases, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.