apt repo: InRelease references a Packages.gz that the CDN serves stale (File has unexpected size 835 != 838)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 125k
- Forks
- 19.4k
- PR merge metrics
- PR metrics pending
Description
What happened
sudo apt update failed against the ChatGPT/Codex apt repo (/etc/apt/sources.list.d/chatgpt.sources) because the published InRelease referenced a Packages.gz that did not match the bytes being served:
Err:12 https://persistent.oaistatic.com/codex-app-prod/linux/deb stable/main amd64 Packages
File has unexpected size (835 != 838). Mirror sync in progress?
Release file created at: Tue, 08 Sep 2026 23:43:45 +0000
E: Failed to fetch .../dists/stable/main/binary-amd64/Packages.gz
E: Some index files failed to download.
Because this is the only failing source, the whole apt update exits non-zero, which breaks unattended upgrades and any CI/provisioning step that runs apt update with set -e.
Diagnosis
At the time of failure:
dists/stable/InRelease(Date: Tue, 08 Sep 2026 23:43:45 +0000) listedmain/binary-amd64/Packages.gzas 838 bytes, sha25691bc5bac6f0fc332ceb586fb032db0047f79b67a6b986f152ce1af7d68c0aae5.- The URL actually served 835 bytes, sha256
0205e26e1e45987079ea8d53a93c3b1de7105dd520c12f7110cb90357647d9c7— a different index, not a truncated one (it gunzipped cleanly to 1523 bytes with a different sha256 than the advertised uncompressedPackages). - The uncompressed
main/binary-amd64/Packageswas already correct and matchedInReleaseexactly (1523 bytes, sha256551e96b9…0147a).
So the plain Packages and the InRelease were in sync, while the .gz variant was a stale generation. Roughly 20 minutes later the .gz served the expected 838 bytes / 91bc5bac… and apt update succeeded, so this looks like an edge-cache or upload-ordering issue that leaves the .gz behind for a window after each publish, rather than a one-off corrupt file.
Suggested fixes
- Publish index files before the
Release/InReleasethat references them, and purge/invalidate the CDN cache fordists/**on publish (or servedists/withCache-Control: no-cache/ short TTL + revalidation). - Make index uploads atomic per publish generation so
PackagesandPackages.gzcan never come from different runs.
Workaround for anyone hitting this
Ask apt to prefer the uncompressed index until the cache catches up (the plain Packages was intact throughout):
sudo apt update -o Acquire::CompressionTypes::Order::=uncompressed
Environment
- Ubuntu 25.10 (resolute), amd64, apt 3.x
- Source:
https://persistent.oaistatic.com/codex-app-prod/linux/deb stable main
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with /etc/apt/sources.list.d/chatgpt.sources and compare dists/stable/InRelease with the served Packages and Packages.gz; the issue provides an uncompressed-index apt update workaround. Trace the apt repository publishing and CDN cache path, then verify that one publish generation is served consistently and apt update succeeds without the workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, ubuntu
- Domain
- devops, infrastructure, release
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100