openai / openai/codex

Desktop (Windows): corrupt logs_2.sqlite B-tree causes a deterministic, unrecoverable app-server handshake timeout

Open
#39,015 3 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

app app-server bug windows-os
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

What version of the Codex App are you using?

26.810.7004.0 — MSIX package OpenAI.Codex_26.810.7004.0_x64
ChatGPT.exe FileVersion 151.0.7922.137, built 2026-08-16

What subscription do you have?

Pro 20x

What platform is your computer?

Microsoft Windows NT 10.0.26200.0 x64

What issue are you seeing?

Codex App failed to launch on every attempt with a blocking modal:

ChatGPT failed to start.
Codex app-server initialize handshake timed out
[ Check for Updates ]  [ Quit ]

Root cause: ~/.codex/logs_2.sqlite had a corrupt B-tree. Not size, not disk space, not a broken install, not lock contention.

Verified by opening the main database file with immutable=1 — WAL bypassed, no locking, no other process participating — so this is not an artifact of the live WAL or of a second app-server holding the file:

PRAGMA quick_check;
*** in database main ***
Tree 5 page 549189 cell 1:  Rowid 341223817 out of order
Tree 5 page 546412 cell 1:  Rowid 341223769 out of order
Tree 5 page 548159 cell 21: Rowid 341224892 out of order

SELECT COUNT(*) FROM logs;  ->  database disk image is malformed

Every other database in CODEX_HOME passed quick_check cleanly, including state_5.sqlite (675 threads). Exactly one file was damaged and it was the non-essential telemetry log — yet it prevented the entire application from starting.

Two log signatures were observed, both from this one cause:

A — majority of launches

initialize_handshake_result durationMs=30012 outcome=failure
  errorMessage="Codex app-server initialize handshake timed out"
app_server_connection.state_changed cause=initialize_handshake_timeout
  connectionError={"code":"restart-required"}
Desktop bootstrap failed to start the main app phase=bootstrap-import-main

B — occasionally, the actionable error surfaced instead

app_server_connection.closed code=1 reason="Error: failed to initialize sqlite
  state runtime under <USERPROFILE>\.codex"
initialize_handshake_result durationMs=5948 outcome=failure

Signature B appeared in only 1 of 6 failed launches. The other five gave no indication a database was involved.

What steps can reproduce the bug?

Preconditions: ~/.codex/logs_2.sqlite has a corrupt B-tree. On this machine it appeared after upgrading OpenAI.Codex 26.707.8479.0 → 26.810.7004.0 (crash dumps from the earlier build are present, dated 2026-07-15). Same origin as #23917.

  1. Launch Codex App.
  2. app-server spawns and begins startup work against the damaged log DB.
  3. At 30.0 s the shell aborts the handshake and shows the modal.
  4. Repeat — identical result, indefinitely.

The key behaviour — the WAL is regenerated each launch, not replayed. This is what distinguishes this from the existing "large WAL" reports:

10:00:34   543,872 B          132 frames      (healthy, pre-launch)
10:08:29   Desktop launched
10:08:37   codex.exe spawned
10:09:15   1,129,127,232 B    274,060 frames  (+1.05 GB in ~38s)
10:09:39   stable — work finished

An earlier failed launch that morning produced 274,306 frames. Two independent runs, ±0.1%. This is deterministic work against a fixed damaged structure, not accumulated history.

The timing margin — it misses by ~2 seconds, every time:

14:08:37.768   transport start — 30s timer begins
14:09:07.783   timeout fires        (30.014s)  -> shell aborts, modal shown
14:09:09.860   codex.exe responds   (32.09s)   -> 2.1s too late

The app-server does complete the handshake. It is killed ~2 s short on every attempt. Because the shell aborts mid-work nothing commits, the damage is never addressed, and the next launch performs identical work. This is why the condition is permanent rather than transient — it structurally cannot self-heal.

Confirmed: checkpointing does NOT fix this. Tested inadvertently — an unrelated checkpoint reduced the WAL to 33 frames; the very next launch regenerated 274,060 frames and failed identically. A corrupt B-tree cannot be checkpointed into health.

Any user can self-diagnose in ~30 seconds, non-destructively:

py -3 -c "import sqlite3,os; p=os.path.expanduser(r'~\.codex\logs_2.sqlite'); c=sqlite3.connect('file:'+p.replace('\\','/')+'?immutable=1',uri=True); print(c.execute('PRAGMA quick_check(3)').fetchall())"

immutable=1 matters: it bypasses the WAL and all locking, so the result reflects the file itself and is unaffected by any running app-server.

Workaround: with the app fully exited (see tray note below) and any IDE Codex extension closed, move logs_2.sqlite, logs_2.sqlite-wal and logs_2.sqlite-shm aside together. The app rebuilds an empty log DB.

before after
logs_2.sqlite 2,439,012,352 B, corrupt 49,152 B, healthy
logs_2.sqlite-wal 274,060 frames ~1,000 frames
handshake 32.1 s vs 30 s → killed window visible in 1.7 s

No chats, memories, threads or settings are lost — none live in this file.

What is the expected behavior?
  1. A corrupt or oversized telemetry log should not be fatal to the application. If state_5.sqlite opens successfully, the app should start. Log-database init/maintenance should not be blocking on the launch path. This alone would have prevented the entire outage.
  2. On detecting SQLITE_CORRUPT in the log DB, rotate it — rename aside, create a fresh one — rather than retrying an operation that cannot succeed. The current behaviour is an infinite loop with a 30-second period.
  3. Run log retention/maintenance asynchronously, after the handshake completes.
  4. Surface the underlying SQLite error in the UI. "Handshake timed out" gave no hint a database was corrupt, and the modal's only options were "Check for Updates" and "Quit" — neither of which can resolve this.
  5. Cap and rotate logs_N.sqlite. 2.27 GB of trace logs is not a useful diagnostic artifact.
  6. Add an integrity check to codex doctor covering every database in CODEX_HOME, so users can self-diagnose without hand-writing SQLite probes.
Additional information

Ruled out, with evidence — recorded to save triage time:

Hypothesis Verdict Evidence
Broken/partial install No Package Status: Ok, Store-signed, binaries present
Application crash No No Event ID 1000 for ChatGPT, no WER report, no crash dump since the prior package version
Disk full No 230 GB free on an NVMe SSD
Permissions / ACL No User holds FullControl on .codex
WSL2 mode (#36320) No Logs show a native Windows spawn from the package path
Outdated Codex CLI No Desktop spawns its own bundled codex.exe from inside the MSIX; cannot be version-skewed with itself
WAL too large to replay No The WAL is written, not read — rebuilt from 132 → 274,060 frames during one failed launch
WAL checkpoint as a fix No Tested; next launch regenerated it and failed identically
Concurrent app-server contention (#30105) No Tested directly — see below
Corrupt user/state data No All other DBs quick_check clean
Reinstalling the app Would not help ~/.codex lives outside the MSIX and survives uninstall entirely

Contention explicitly tested and excluded. An IDE extension ran its own codex.exe app-server against the same CODEX_HOME throughout. After the fix this was tested in the harsher ordering — IDE app-server started first and already holding the databases, then Desktop launched:

codex pid A — IDE extension's,  started 10:41:51
codex pid B — Desktop's,        started 10:42:59

14:42:59.652  codex.exe spawned
14:43:00.016  Transport start success   +0.36s
14:43:01.223  window ready-to-show      +1.70s

Two app-servers coexisted without incident.

Integrity of every other database (quick_check via immutable=1, none modified at any point):

Database Size Result
state_5.sqlite 50.7 MB OK (675 threads)
memories_1.sqlite 40 KB OK
goals_1.sqlite 32 KB OK
queue_1.sqlite 28 KB OK
sqlite/codex-dev.db 0.3 MB OK
sqlite/codex-thread-summaries-dev.db 12 KB OK
logs_2.sqlite 2.27 GB CORRUPT

Affected schema (diagnostics only — no user-authored records):

CREATE TABLE logs (
  id INTEGER PRIMARY KEY AUTOINCREMENT, ts INTEGER NOT NULL,
  ts_nanos INTEGER NOT NULL, level TEXT NOT NULL, target TEXT NOT NULL,
  feedback_log_body TEXT, module_path TEXT, file TEXT, line INTEGER,
  thread_id TEXT, process_uuid TEXT, estimated_bytes INTEGER NOT NULL DEFAULT 0
);
-- plus idx_logs_ts, idx_logs_thread_id, idx_logs_thread_id_ts,
--      idx_logs_process_uuid_threadless_ts

File state: 2,439,012,352 B = 595,462 pages × 4096, freelist 539, journal_mode=wal.

Secondary observations

  1. WAL sidecars persist across clean exit. After a proper tray-quit with zero remaining processes, all -wal/-shm files remained on disk. This looks intentional (PERSIST_WAL), but it means the presence of sidecar files is not evidence of an unclean shutdown. Several wrong conclusions were reached during this investigation by assuming otherwise — documenting it so it doesn't mislead others.

  2. The app stays resident after the window is closed. Closing the window left 10 processes and the codex.exe app-server running, still holding every database. "I closed Codex App" and "Codex App is not running" are different states. Anyone attempting a file-level workaround must quit via the system tray or they will be operating on open files.

  3. Startup shell-environment resolution times out — not causal, but it delays every launch by ~7 s:

    Failed to load shell env  caller=startup  status=timed_out
      timeoutMs=5000  durationMs=7144  pathEntryCount=30
    

    PATH contained several duplicated system entries. This happens before the app-server spawn, so it sits outside the 30 s handshake window — but it is real overhead and would erode the margin on a machine already close to the limit.

Related issues

  • #23917 (closed) — closest match and the likely origin. This machine had just been upgraded 26.707.8479.0 → 26.810.7004.0. Suggest reopening or linking; the upgrade path may be where the corruption is introduced.
  • #27741 (open) — same symptom and same workaround, attributed to file size. This report suggests corruption is the actual mechanism in at least some of those cases — worth asking reporters there for a quick_check before chasing size thresholds.
  • #28997, #24275, #31142 (open) — unbounded WAL growth and high-frequency TRACE/streaming writes. Together, likely why the file grew large enough to be at risk in the first place.
  • #30105 (open) — concurrent app-server contention. Investigated and explicitly excluded here by direct experiment.
  • #22444 (closed) — stale processes retaining deleted WALs. Relevant caution for anyone applying the workaround.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the app-server startup path and SQLite initialization for CODEX_HOME, focusing on logs_2.sqlite and the codex doctor entry point. Use the supplied immutable quick_check probe to reproduce the corruption case. Done means a corrupt or oversized log database no longer blocks the handshake, the underlying error is surfaced, and integrity checking covers the databases in CODEX_HOME.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sqlite
Domain
databases, desktop
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.