openai / openai/codex

apt repo: InRelease references a Packages.gz that the CDN serves stale (File has unexpected size 835 != 838)

Open
#43,968 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

app bug Linux
Dominant language
Rust
Stars
125k
Forks
19.4k
PR merge metrics
PR metrics pending

Description

What happened

sudo apt update failed against the ChatGPT/Codex apt repo (/etc/apt/sources.list.d/chatgpt.sources) because the published InRelease referenced a Packages.gz that did not match the bytes being served:

Err:12 https://persistent.oaistatic.com/codex-app-prod/linux/deb stable/main amd64 Packages
  File has unexpected size (835 != 838). Mirror sync in progress?
  Release file created at: Tue, 08 Sep 2026 23:43:45 +0000
E: Failed to fetch .../dists/stable/main/binary-amd64/Packages.gz
E: Some index files failed to download.

Because this is the only failing source, the whole apt update exits non-zero, which breaks unattended upgrades and any CI/provisioning step that runs apt update with set -e.

Diagnosis

At the time of failure:

  • dists/stable/InRelease (Date: Tue, 08 Sep 2026 23:43:45 +0000) listed main/binary-amd64/Packages.gz as 838 bytes, sha256 91bc5bac6f0fc332ceb586fb032db0047f79b67a6b986f152ce1af7d68c0aae5.
  • The URL actually served 835 bytes, sha256 0205e26e1e45987079ea8d53a93c3b1de7105dd520c12f7110cb90357647d9c7 — a different index, not a truncated one (it gunzipped cleanly to 1523 bytes with a different sha256 than the advertised uncompressed Packages).
  • The uncompressed main/binary-amd64/Packages was already correct and matched InRelease exactly (1523 bytes, sha256 551e96b9…0147a).

So the plain Packages and the InRelease were in sync, while the .gz variant was a stale generation. Roughly 20 minutes later the .gz served the expected 838 bytes / 91bc5bac… and apt update succeeded, so this looks like an edge-cache or upload-ordering issue that leaves the .gz behind for a window after each publish, rather than a one-off corrupt file.

Suggested fixes
  • Publish index files before the Release/InRelease that references them, and purge/invalidate the CDN cache for dists/** on publish (or serve dists/ with Cache-Control: no-cache / short TTL + revalidation).
  • Make index uploads atomic per publish generation so Packages and Packages.gz can never come from different runs.
Workaround for anyone hitting this

Ask apt to prefer the uncompressed index until the cache catches up (the plain Packages was intact throughout):

sudo apt update -o Acquire::CompressionTypes::Order::=uncompressed
Environment
  • Ubuntu 25.10 (resolute), amd64, apt 3.x
  • Source: https://persistent.oaistatic.com/codex-app-prod/linux/deb stable main

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with /etc/apt/sources.list.d/chatgpt.sources and compare dists/stable/InRelease with the served Packages and Packages.gz; the issue provides an uncompressed-index apt update workaround. Trace the apt repository publishing and CDN cache path, then verify that one publish generation is served consistently and apt update succeeds without the workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, ubuntu
Domain
devops, infrastructure, release
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.