SQLite disk I/O errors and unusable rollout files when CODEX_HOME is on CERN AFS

What version of Codex CLI is running?

codex-cli 0.153.4

What subscription do you have?

ChatGPT Plus

Which model were you using?

gpt-5.6-terra

What platform is your computer?

Linux 5.14.0-687.41.1.el9_8.x86_64 x86_64 x86_64

What terminal emulator and version are you using (if applicable)?

VS Code

Codex doctor report

[yet@lxplus953 realdata_fit]$ codex doctor --json
{
  "schemaVersion": 1,
  "generatedAt": "1789009207s since unix epoch",
  "overallStatus": "warning",
  "codexVersion": "0.154.0",
  "checks": {
    "app_server.status": {
      "id": "app_server.status",
      "category": "app-server",
      "status": "ok",
      "summary": "background server is not running",
      "details": {
        "control socket": "/afs/cern.ch/user/y/yet/.codex/app-server-control/app-server-control.sock",
        "daemon state dir": "/afs/cern.ch/user/y/yet/.codex/app-server-daemon",
        "mode": "ephemeral",
        "pid file": "/afs/cern.ch/user/y/yet/.codex/app-server-daemon/app-server.pid (missing)",
        "settings": "/afs/cern.ch/user/y/yet/.codex/app-server-daemon/settings.json (missing)",
        "status": "not running",
        "update-loop pid file": "/afs/cern.ch/user/y/yet/.codex/app-server-daemon/app-server-updater.pid (missing)"
      },
      "remediation": null,
      "durationMs": 22
    },
    "auth.credentials": {
      "id": "auth.credentials",
      "category": "auth",
      "status": "ok",
      "summary": "auth is configured",
      "details": {
        "auth file": "/afs/cern.ch/user/y/yet/.codex/auth.json",
        "auth storage mode": "File",
        "stored API key": "false",
        "stored ChatGPT tokens": "true",
        "stored agent identity": "false",
        "stored auth mode": "chatgpt"
      },
      "remediation": null,
      "durationMs": 0
    },
    "config.load": {
      "id": "config.load",
      "category": "config",
      "status": "ok",
      "summary": "config loaded",
      "details": {
        "CODEX_HOME": "/afs/cern.ch/user/y/yet/.codex",
        "active thread overrides": "not inspected",
        "config.toml": "/afs/cern.ch/user/y/yet/.codex/config.toml",
        "config.toml parse": "ok",
        "configuration load ms": "1446",
        "configuration scope": "invocation config, including cloud-managed policy",
        "cwd": "/afs/cern.ch/work/y/yet/private/pbar_ihep/realdata_fit",
        "enabled feature flags": "shell_tool, view_image, sleep_tool, unified_exec, unified_exec_tty, unified_exec_zsh_fork, shell_snapshot, content_item_kinds, code_mode_host, terminal_resize_reflow, sqlite, hooks, enable_request_compression, unbounded_connection_retries, multi_agent, apps, tool_search_always_defer_mcp_tools, tool_suggest, plugins, in_app_browser, in_app_chat, in_app_dictation, in_app_local_automation, in_app_updates, browser_use, browser_use_full_cdp_access, browser_use_external, computer_use, remote_plugin, plugin_sharing, image_generation, resize_all_images, item_ids, skill_mcp_dependency_install, skill_search, mentions_v2, steer, guardian_approval, goals, collaboration_modes, tool_call_mcp_elicitation, auth_elicitation, personality, fast_mode, tui_app_server, remote_compaction_v2, compaction_image_budget, workspace_dependencies",
        "feature flag overrides": "none",
        "feature flags enabled": "48",
        "log dir": "/afs/cern.ch/user/y/yet/.codex/log",
        "mcp servers": "0",
        "model": "gpt-5.6-terra",
        "model provider": "openai",
        "sqlite home": "/afs/cern.ch/user/y/yet/.codex"
      },
      "remediation": null,
      "durationMs": 0
    },
    "git.environment": {
      "id": "git.environment",
      "category": "git",
      "status": "ok",
      "summary": "git executable found; execution not verified",
      "details": {
        "PATH git #1": "/usr/bin/git",
        "PATH git entries": "1",
        "git execution": "not inspected (PATH helpers are not executed)",
        "repo detected": "false",
        "selected git": "/usr/bin/git"
      },
      "remediation": null,
      "durationMs": 0
    },
    "installation": {
      "id": "installation",
      "category": "install",
      "status": "ok",
      "summary": "installation looks consistent",
      "details": {
        "PATH codex #1": "/afs/cern.ch/user/y/yet/.local/npm/bin/codex",
        "current executable": "/afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin/codex",
        "install context": "npm (package /afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl, bin /afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin, resources /afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-resources, path /afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-path)",
        "managed by Vite+": "false",
        "managed by bun": "false",
        "managed by npm": "true",
        "managed by pnpm": "false",
        "managed package root": "/afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex",
        "npm update target": "not inspected (PATH helpers are not executed)"
      },
      "remediation": null,
      "durationMs": 17
    },
    "mcp.config": {
      "id": "mcp.config",
      "category": "mcp",
      "status": "ok",
      "summary": "no MCP servers configured",
      "details": {},
      "remediation": null,
      "durationMs": 0
    },
    "network.env": {
      "id": "network.env",
      "category": "network",
      "status": "ok",
      "summary": "network-related environment looks readable",
      "details": {
        "managed proxy": "not configured",
        "proxy env vars": "none",
        "respect system proxy": "disabled"
      },
      "remediation": null,
      "durationMs": 0
    },
    "network.provider_reachability": {
      "id": "network.provider_reachability",
      "category": "reachability",
      "status": "ok",
      "summary": "active provider endpoints are reachable over HTTP",
      "details": {
        "ChatGPT inference URL": "https://chatgpt.com/backend-api/<redacted> reachable (HTTP 405)",
        "reachability mode": "ChatGPT auth"
      },
      "remediation": null,
      "durationMs": 701
    },
    "network.websocket_reachability": {
      "id": "network.websocket_reachability",
      "category": "websocket",
      "status": "ok",
      "summary": "Responses WebSocket handshake succeeded",
      "details": {
        "DNS": "2 IPv4, 2 IPv6, first IPv6",
        "auth mode": "chatgpt",
        "connect timeout": "15000 ms",
        "endpoint": "wss://chatgpt.com/backend-api/<redacted>",
        "handshake result": "HTTP 101 Switching Protocols",
        "model provider": "openai",
        "provider name": "OpenAI",
        "proxy env vars": "none",
        "reasoning header": "false",
        "server model present": "false",
        "supports websockets": "true",
        "wire API": "responses"
      },
      "remediation": null,
      "durationMs": 1589
    },
    "runtime.provenance": {
      "id": "runtime.provenance",
      "category": "runtime",
      "status": "ok",
      "summary": "running npm on linux-x86_64",
      "details": {
        "commit": "unknown",
        "current executable": "/afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin/codex",
        "install method": "npm (package /afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl, bin /afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/bin, resources /afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-resources, path /afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-path)",
        "platform": "linux-x86_64",
        "version": "0.154.0"
      },
      "remediation": null,
      "durationMs": 0
    },
    "runtime.search": {
      "id": "runtime.search",
      "category": "search",
      "status": "ok",
      "summary": "search command found (bundled); execution not verified",
      "details": {
        "search command": "/afs/cern.ch/user/y/yet/.local/npm/lib/node_modules/@openai/codex/node_modules/@openai/codex-linux-x64/vendor/x86_64-unknown-linux-musl/codex-path/rg",
        "search command readiness": "file exists",
        "search provider": "bundled"
      },
      "remediation": null,
      "durationMs": 0
    },
    "sandbox.helpers": {
      "id": "sandbox.helpers",
      "category": "sandbox",
      "status": "ok",
      "summary": "sandbox configuration is readable",
      "details": {
        "approval policy": "OnRequest",
        "codex-linux-sandbox helper": "/afs/cern.ch/user/y/yet/.codex/tmp/arg0/codex-arg0brmGbi/codex-linux-sandbox",
        "denied-read glob rules": "0",
        "denied-read rules": "0",
        "execve wrapper helper": "/afs/cern.ch/user/y/yet/.codex/tmp/arg0/codex-arg0brmGbi/codex-execve-wrapper",
        "filesystem sandbox": "restricted",
        "glob scan max depth": "unbounded",
        "managed filesystem source": "none",
        "network sandbox": "restricted"
      },
      "remediation": null,
      "durationMs": 8
    },
    "security.endpoint": {
      "id": "security.endpoint",
      "category": "security",
      "status": "ok",
      "summary": "endpoint protection is not inspected on this platform",
      "details": {
        "endpoint products": "not inspected on this platform"
      },
      "remediation": null,
      "durationMs": 16
    },
    "state.paths": {
      "id": "state.paths",
      "category": "state",
      "status": "ok",
      "summary": "state paths and databases are inspectable",
      "details": {
        "CODEX_HOME": "/afs/cern.ch/user/y/yet/.codex (dir)",
        "active rollout files": "9 files, 29651197 total bytes, 3294577 average bytes",
        "archived rollout files": "0 files, 0 total bytes, 0 average bytes",
        "goals DB": "/afs/cern.ch/user/y/yet/.codex/goals_1.sqlite (file)",
        "goals DB integrity": "ok",
        "log DB": "/afs/cern.ch/user/y/yet/.codex/logs_2.sqlite (file)",
        "log DB integrity": "ok",
        "log dir": "/afs/cern.ch/user/y/yet/.codex/log (missing)",
        "memories DB": "/afs/cern.ch/user/y/yet/.codex/memories_1.sqlite (file)",
        "memories DB integrity": "ok",
        "queue DB": "/afs/cern.ch/user/y/yet/.codex/queue_1.sqlite (file)",
        "queue DB integrity": "ok",
        "sqlite home": "/afs/cern.ch/user/y/yet/.codex (dir)",
        "state DB": "/afs/cern.ch/user/y/yet/.codex/state_5.sqlite (file)",
        "state DB integrity": "ok",
        "thread history DB": "/afs/cern.ch/user/y/yet/.codex/thread_history_1.sqlite (file)",
        "thread history DB integrity": "ok"
      },
      "remediation": null,
      "durationMs": 2835
    },
    "state.rollout_db_parity": {
      "id": "state.rollout_db_parity",
      "category": "threads",
      "status": "warning",
      "summary": "rollout files and state DB thread inventory differ",
      "details": {
        "default model provider": "openai",
        "rollout DB active files": "6",
        "rollout DB active rows": "13",
        "rollout DB archive mismatches": "0",
        "rollout DB archived files": "0",
        "rollout DB archived rows": "0",
        "rollout DB duplicate DB paths": "0",
        "rollout DB duplicate rollout thread ids": "0",
        "rollout DB malformed file names": "0",
        "rollout DB missing active rows": "0",
        "rollout DB missing archived rows": "0",
        "rollout DB model providers": "openai=13",
        "rollout DB rows": "13",
        "rollout DB scan cap reached": "false",
        "rollout DB scan error sample": [
          "/afs/cern.ch/user/y/yet/.codex/sessions/2026/09/09/rollout-2026-09-09T14-19-07-01a0861b-8c8a-73f3-b4ec-8315199266e0.jsonl (rollout at /afs/cern.ch/user/y/yet/.codex/sessions/2026/09/09/rollout-2026-09-09T14-19-07-01a0861b-8c8a-73f3-b4ec-8315199266e0.jsonl has no usable header record)",
          "/afs/cern.ch/user/y/yet/.codex/sessions/2026/09/08/rollout-2026-09-08T07-21-30-01a07f76-d900-7de2-8b9c-527e5986bbe3.jsonl (rollout at /afs/cern.ch/user/y/yet/.codex/sessions/2026/09/08/rollout-2026-09-08T07-21-30-01a07f76-d900-7de2-8b9c-527e5986bbe3.jsonl has no usable header record)",
          "/afs/cern.ch/user/y/yet/.codex/sessions/2026/09/04/rollout-2026-09-04T06-03-14-01a06a95-c30f-76b3-a441-4fde683a5575.jsonl (rollout at /afs/cern.ch/user/y/yet/.codex/sessions/2026/09/04/rollout-2026-09-04T06-03-14-01a06a95-c30f-76b3-a441-4fde683a5575.jsonl has no usable header record)"
        ],
        "rollout DB scan errors": "3",
        "rollout DB sources": "subagent:other=7, cli=6",
        "rollout DB stale row sample": [
          "/afs/cern.ch/user/y/yet/.codex/sessions/2026/09/03/rollout-2026-09-03T04-44-39-01a06527-731f-7762-afd3-7c47ef80c152.jsonl",
          "/afs/cern.ch/user/y/yet/.codex/sessions/2026/09/03/rollout-2026-09-03T04-55-46-01a06531-a2fb-7950-a3b9-16051e4fde5d.jsonl",
          "/afs/cern.ch/user/y/yet/.codex/sessions/2026/09/03/rollout-2026-09-03T07-15-43-01a065b1-c351-7c12-a8b1-fc98dce4bf97.jsonl",
          "/afs/cern.ch/user/y/yet/.codex/sessions/2026/09/03/rollout-2026-09-03T07-15-44-01a065b1-c644-7753-ba2e-6a9e0ad4a62c.jsonl"
        ],
        "rollout DB stale rows": "4"
      },
      "issues": [
        {
          "severity": "warning",
          "cause": "state DB rows point at missing or unusable rollout files",
          "measured": "4 stale rows",
          "expected": "every state DB rollout path is a file on disk",
          "remedy": null,
          "fields": []
        },
        {
          "severity": "warning",
          "cause": "rollout scan was incomplete or found bad files",
          "measured": "3 scan errors, 0 malformed names, scan cap reached: false",
          "expected": "rollout directories are fully scannable",
          "remedy": "Check file permissions and unexpected files under CODEX_HOME sessions.",
          "fields": []
        }
      ],
      "remediation": null,
      "durationMs": 1816
    },
    "system.disk": {
      "id": "system.disk",
      "category": "disk",
      "status": "ok",
      "summary": "sufficient free disk space (2048.0 GiB)",
      "details": {
        "CODEX_HOME available": "2048.0 GiB",
        "failure threshold": "1.0 GiB",
        "warning threshold": "5.0 GiB",
        "worktree available": "2048.0 GiB"
      },
      "remediation": null,
      "durationMs": 0
    },
    "system.environment": {
      "id": "system.environment",
      "category": "system",
      "status": "ok",
      "summary": "OS language en-US",
      "details": {
        "EDITOR": "set",
        "LANG": "en_US.UTF-8",
        "VISUAL": "not set",
        "os": "Red Hat Enterprise Linux 9.8.0 [64-bit]",
        "os language": "en-US",
        "os type": "Red Hat Enterprise Linux",
        "os version": "9.8.0"
      },
      "remediation": null,
      "durationMs": 95
    },
    "terminal.env": {
      "id": "terminal.env",
      "category": "terminal",
      "status": "warning",
      "summary": "width 67 cols - output may wrap (recommended >=80)",
      "details": {
        "SSH_CLIENT": "present",
        "SSH_CONNECTION": "present",
        "SSH_TTY": "present",
        "TERM": "xterm-256color",
        "color output": "enabled",
        "effective locale": "en_US.UTF-8",
        "stderr is terminal": "true",
        "stdin is terminal": "true",
        "stdout is terminal": "true",
        "terminal": "unknown",
        "terminal size": "67x27"
      },
      "issues": [
        {
          "severity": "warning",
          "cause": "width 67 cols - output may wrap (recommended >=80)",
          "measured": "67 x 27",
          "expected": ">= 80 columns",
          "remedy": "resize the window to at least 80 columns",
          "fields": [
            "terminal size"
          ]
        }
      ],
      "remediation": null,
      "durationMs": 0
    },
    "terminal.title": {
      "id": "terminal.title",
      "category": "title",
      "status": "ok",
      "summary": "terminal title default",
      "details": {
        "terminal title activity": "true",
        "terminal title items": "activity, project-name",
        "terminal title project source": "project config",
        "terminal title project value": "realdata_fit",
        "terminal title source": "default"
      },
      "remediation": null,
      "durationMs": 0
    },
    "updates.status": {
      "id": "updates.status",
      "category": "updates",
      "status": "ok",
      "summary": "update configuration is locally consistent",
      "details": {
        "cached latest version": "0.154.0",
        "check for update on startup": "true",
        "last checked at": "2026-09-10T02:12:09.495409020Z",
        "latest version": "0.154.0",
        "latest version status": "current version is not older",
        "npm update target": "not inspected (PATH helpers are not executed)",
        "update action": "npm install -g @openai/codex",
        "version cache": "/afs/cern.ch/user/y/yet/.codex/version.json"
      },
      "remediation": null,
      "durationMs": 1203
    }
  }
}

What issue are you seeing?

• CERN lxplus
• Red Hat Enterprise Linux 9.8
• Codex CLI 0.153.4
• CODEX_HOME=~/.codex
• ~/.codex is stored on CERN AFS
• AFS network filesystem
• Codex is accessed over SSH from different lxplus hosts

The startup failure contains:

failed to initialize state runtime at <CODEX_HOME>:
failed to open log DB at <CODEX_HOME>/logs_2.sqlite:
error returned from database: (code: 522) disk I/O error

codex doctor may report the main SQLite databases as having valid integrity:

state_5.sqlite integrity ok
logs_2.sqlite integrity ok
goals_1.sqlite integrity ok
memories_1.sqlite integrity ok
queue_1.sqlite integrity ok

while operations on another runtime database, such as thread_history_1.sqlite, fail with:

(code: 522) disk I/O error

There is also a session persistence problem that appears related.

After an abnormal SSH disconnect, terminal closure, or network interruption, some rollout JSONL files can become unusable. codex doctor reports errors such as:

rollout ... has no usable header record

I have also observed a session where the conversation continued normally beyond a certain point, and Codex appeared to retain the later conversational context during that running process. However, after exiting/reconnecting and running codex resume, the visible restored conversation stopped at an earlier message and the later turns were missing.

This makes it appear that the runtime/database state and the persisted rollout JSONL can become inconsistent when Codex is running with CODEX_HOME on AFS.

Disk space is sufficient, and network/authentication checks otherwise pass.

What steps can reproduce the bug?

The issue is intermittent, but the sequence in which I have observed it is:

Use a Codex home directory stored on CERN AFS:

echo "$HOME"

/afs/cern.ch/user//...

with the default:

CODEX_HOME="$HOME/.codex"

Start Codex on a CERN lxplus machine over SSH:

codex

Have a normal multi-turn conversation.

Terminate the SSH session abnormally, for example because of a network interruption or by closing the terminal without cleanly exiting Codex.

Reconnect to lxplus. The new SSH session may be assigned to a different lxplus host, while the same AFS-backed home directory is used.

Run:

codex

or:

codex resume

In some cases Codex fails during SQLite initialization with:

(code: 522) disk I/O error

Running:

codex doctor

may show SQLite integrity checks as ok for some databases but an I/O error for another database.

It may also report rollout files with:

has no usable header record

In another observed failure mode, codex resume restores only the earlier part of a conversation even though the original running session had continued beyond that point.

I do not currently have a deterministic minimal reproduction because the problem appears to depend on interrupted writes and/or AFS filesystem behavior.

What is the expected behavior?

Codex should not leave its local state or session history unusable after an SSH or network interruption.

Ideally, one or more of the following should happen:

Codex should work correctly when CODEX_HOME is located on AFS or another supported network filesystem.

If AFS filesystem semantics are incompatible with the SQLite configuration used by Codex, Codex should detect this and provide a clear warning rather than later failing with a generic SQLite disk I/O error.

Runtime SQLite files should be configurable separately from persistent Codex data, allowing SQLite databases to live on a host-local filesystem while session/configuration data remain on shared storage.

If the runtime SQLite databases become unusable but the rollout files are valid, Codex should be able to rebuild or re-index the runtime state automatically.

Rollout JSONL writes should remain recoverable after abnormal process or SSH termination, without silently losing later conversation turns.

Additional information

This appears potentially related to existing reports involving SQLite on filesystems whose semantics differ from native local filesystems.

For example, there is an existing report where SQLite databases located on WSL DrvFs fail with disk I/O error, while the same databases work correctly after being copied to a native ext4 filesystem.

There is also an existing issue discussing a shared CODEX_HOME with host-local SQLite storage through CODEX_SQLITE_HOME.

However, I could not find an existing issue specifically covering CERN AFS together with both:

SQLite disk I/O errors

and:

rollout JSONL files becoming unusable / incomplete after interrupted SSH sessions

The problem therefore may involve SQLite locking/WAL/shared-memory/filesystem synchronization semantics on AFS, interrupted rollout writes, or both.

I am happy to provide additional filesystem or SQLite diagnostics if useful.

Start by reproducing the reported failure with CODEX_HOME on CERN AFS, using the SQLite state databases and active rollout files described in the doctor report. Trace the disk I/O and rollout parity behavior, then verify that SQLite operations succeed and rollout files remain usable in that environment.