Shared models_cache.json oscillates between originator-specific catalogs across app-server processes

Open
#33,593 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
35/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Quiet
Tech stack
rust

Research direction

Start at the model manager and app-server refresh worker around /models, CODEX_HOME/models_cache.json, and cache identity construction. Reproduce the two-originator case with shared temporary CODEX_HOME, alternating mocked catalogs, process recreation, and offline lookup. Done means each originator retains and recovers its own catalog, while concurrent writes remain complete and parseable.

Written by the indexing model from the issue text.

Description

app-server bug
What version of Codex CLI is running?

codex-cli 0.144.5

What subscription do you have?

ChatGPT authentication. The exact subscription tier is not relevant to the
reproduction.

Which model were you using?

gpt-5.6-sol

What platform is your computer?

macOS 26.5.2, Darwin 25.5.0, Apple Silicon.

What issue are you seeing?

Two long-lived app-server processes built from the exact same Codex binary and using the same CODEX_HOME, provider, auth mode, and ChatGPT account receive different model catalogs when only their app-server originator differs.

Both processes persist their originator-specific response to the same global $CODEX_HOME/models_cache.json. The file therefore alternates between two
valid, but semantically incompatible, catalogs:

  • Codex Desktop / codex_cli_rs: gpt-5.6-sol has
    multi_agent_version = v1 and tool_mode = direct.
  • codex-chrome-extension-sidepanel: the same model has
    multi_agent_version = v2 and tool_mode = code_mode_only.

The app-server background refresh worker repeats the overwrite approximately
every three minutes. Depending on which process wrote last, newly started
sessions and runtime metadata consumers can see a different multi-agent/tool
configuration.

This is not a corrupt server response and is not explained by a CLI version,
provider, or account mismatch. It is a cache-identity and cross-process storage
problem.

Confirmed A/B measurements

The two app-server processes had:

  • the same executable SHA-256;
  • Codex CLI 0.144.5;
  • the same CODEX_HOME, auth file, provider, token, account, request URL, and
    User-Agent;
  • different app-server client identities/originators.

Direct /models?client_version=0.144.5 requests changing only originator
returned:

Originator Models ETag Sol multi-agent Sol tool mode
Codex Desktop W/"6cbd9267ee12889ea6614486cc5879ff" v1 direct
codex_cli_rs W/"6cbd9267ee12889ea6614486cc5879ff" v1 direct
codex-chrome-extension-sidepanel W/"68e28197d956a749f12435108b07ff25" v2 code_mode_only

Observed cache writes included:

Time (UTC) Writer Request ID Result
11:33:36 Codex Desktop 25cc13a9-0991-4bc8-a5e6-ed9a125bd759 V1/direct catalog written
11:34:03 Chrome sidepanel bb12724f-9a1d-4ef3-a256-1fa3c9fe3b1e V2/code-mode catalog written
about 3 minutes later Chrome sidepanel cfaa5c42-34e5-4600-8e1e-73df23bb9132 the alternating pattern repeated

No token, account identifier, raw session transcript, or auth file is included
in this report.

What steps can reproduce the bug?
Reproduction A: two app-server clients
  1. Use one Codex binary, one CODEX_HOME, and one ChatGPT account.
  2. Start app-server A and initialize it with
    clientInfo.name = "Codex Desktop".
  3. Start app-server B and initialize it with
    clientInfo.name = "codex-chrome-extension-sidepanel".
  4. Request the model catalog from both processes.
  5. Record the request originator, ETag, and the gpt-5.6-sol fields
    multi_agent_version, tool_mode, and use_responses_lite.
  6. Observe that the server returns different ETags/catalog metadata.
  7. Observe both processes writing to the same
    $CODEX_HOME/models_cache.json.
  8. Wait for multiple app-server refresh intervals and observe the file
    alternating again.
Reproduction B: Desktop plus Chrome extension
  1. Fully quit ChatGPT Desktop and Chrome.
  2. Start ChatGPT Desktop and create a Codex task.
  3. Record app-server ancestry, initialize identity, /models originator, ETag,
    and the cache hash/content.
  4. Start Chrome and open the ChatGPT extension side panel.
  5. Confirm that the extension's native host starts another Codex app-server.
  6. Record the same data and monitor models_cache.json.
  7. Close only the side panel, then Chrome, and repeat to determine the precise
    native-host lifecycle.

In the observed run, the second app-server was a child of the ChatGPT for
Chrome native extension host, not a browser tab merely visiting chatgpt.com.

Reproduction C: deterministic regression test

Use two model managers sharing one temporary CODEX_HOME. Mock /models so
the same provider/account/version returns catalog A for originator A and
catalog B for originator B. Alternate online refreshes, drop both managers,
recreate them offline, and verify that each surface can recover its own
catalog. Current single-file behavior cannot preserve both.

Root cause

The request identity and cache identity do not match.

/models uses the app-server originator, and the service can return a different
catalog and ETag for each client surface. The persisted cache is nevertheless a
single global file. Cache eligibility on current main does not include
originator/client surface, and the file is last-writer-wins.

There are three interacting layers:

  1. Semantic isolation: originator-specific catalogs share one cache slot.
  2. Cross-process coordination: independent app-servers have separate
    in-memory ETag/catalog state but overwrite the same disk file.
  3. Write safety/amplification: the cache is rewritten as a complete file,
    including matching-ETag TTL renewal, without a per-identity slot. Related
    reports also show truncated/EOF reads and excessive write churn.

The app-server refresh worker makes the latent design issue persistent: each
long-lived process refreshes immediately and then approximately every three
minutes.

How the code reached this state
  • #7722 introduced one disk cache, a five-minute TTL, and ETag storage.
  • #8491 added refresh on a model ETag mismatch.
  • #8873 and #8988 made app-server clientInfo.name the process originator,
    allowing /models requests to represent the actual client surface.
  • #9174 renews cache freshness on matching ETags by rewriting the cache.
  • #10414 rejects caches from another client version, but does not preserve
    multiple producer identities.
  • #18950 moved model discovery under providers. A Codex P1 review explicitly
    warned that cache eligibility needed provider identity or separate cache
    files. The PR was subsequently approved and merged without adding that
    isolation, leaving the single-slot design debt in place:
    https://github.com/openai/codex/pull/18950#pullrequestreview-4158982727
  • #28699 added immediate and three-minute app-server background refreshes to
    avoid cold model-list latency. That change did not create the originator
    split, but it turns occasional last-writer-wins contamination into a
    recurring oscillation.
  • #30984 is a current draft that adds provider_id, auth_mode, and
    account_id to cache eligibility. It does not include originator and still
    uses a single file, so both producers in this reproduction have the same key
    and one surface's catalog still evicts the other's.

Related reports:

  • #32482 and #33146: a different runtime/version overwrites the shared cache.
  • #32496: matching-ETag TTL renewal causes high write volume.
  • #30864: truncated/EOF cache reads.
  • #32850: a running UI and a fresh catalog disagree.
  • #33575 and #33547: model-visible MCP tools disappear after runtime metadata
    changes.
  • #32086: a separate downstream bug where Responses Lite makes deferred V1
    tools unreachable because tool_search is not exposed correctly.

The last distinction matters: cache isolation explains why a surface/session
can unexpectedly receive V1 versus V2 metadata. If the server intentionally
returns V1/Responses Lite to a surface, #32086 still needs its own fix.

What is the expected behavior?

Different catalog identities must not overwrite or reuse one another's model
metadata. A client surface should consistently receive and recover the catalog
that was fetched for its own request identity, including after process restart
or while temporarily offline.

Suggested fix

Treat the catalog scope as a first-class identity derived from the same
snapshot used to make the request:

provider_id
endpoint identity/fingerprint
effective auth mode
account/principal
exact originator/client surface
client-version compatibility

The smallest robust storage change is independent per-identity cache slots,
preferably separate files keyed by a hash of that tuple. Each slot should own
its catalog, ETag, and freshness. Storing the account ID directly in the
filename is unnecessary.

Also recommended:

  • snapshot auth and originator once per fetch, then use that snapshot for both
    the HTTP request and cache key;
  • start the refresh worker only after the app-server initialize identity is
    fixed;
  • use atomic temporary-file replacement;
  • use a same-key cross-process lock or another conflict-safe strategy;
  • avoid rewriting the full catalog for every equal-ETag freshness renewal,
    for example by using a small freshness sidecar or coalescing writes;
  • keep in-memory catalog/ETag state partitioned by the same identity.

Adding only an originator validation field to the existing single file would prevent applying a mismatched catalog, but it would still cause eviction, repeated network fetches, and loss of the previous surface's offline state.

Validation plan

Please include regression coverage for:

  1. Same binary/provider/account/version, two originators, different ETags/catalogs.
  2. Alternating online writes in one CODEX_HOME.
  3. Dropping and recreating both managers, followed by offline lookup for each originator.
  4. Multiple background-worker cycles.
  5. Concurrent writers always leaving parseable, complete JSON.
  6. Failure injection during atomic replacement.
  7. Matching-ETag renewal affecting only the correct identity and obeying a write-rate bound.
  8. Initialization races where a worker starts before client identity is known.
  9. A separate #32086 regression proving that V1/Responses Lite retains a usable top-level tool_search route.
Why this is a separate root-cause report

Existing issues cover symptoms or adjacent dimensions: V1 tool serialization, MCP tools disappearing, version skew, provider/account contamination, write amplification, and truncated files. This reproduction isolates a missing cache dimension that is not covered by those reports: the exact same binary, provider, auth mode, account, and client version receive different catalogs
solely because their originators differ, then continuously overwrite one shared cache file.

Dominant language
Rust
Stars
125k
Forks
19.5k
Avg merge
1m
Merged PRs (30d)
1k

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from openai/codex

All issues in openai/codex

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.