WebAssembly / WebAssembly/WASI

Hosting WITs via OCI on GHCR is flaky

Open
#886 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
5.8k
Forks
333
Avg merge
2d 13h
Merged PRs (30d)
3

Description

Today we did a security release of Wasmtime which involves doing a lot of CI all at once. We had lots of flaky failures due to wkg being unable to download WITs, such as:

+ wkg get --format wit --overwrite wasi:cli@0.2.6 -o crates/wasi/src/p2/wit/deps/cli.wit
Getting wasi:cli@0.2.6...
2026-02-24T17:57:34.381595Z  WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
2026-02-24T17:57:34.595113Z  WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
2026-02-24T17:57:34.946847Z  WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
2026-02-24T17:57:36.054084Z  WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
Error: registry error: HTTP status server error (503 Egress is over the account limit.) for url (https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:fdbe84136b3dd46d94305ef37f24f3cf04a70cc2026dca2592ac2ec0c9de15c7?se=2026-02-24T18%3A05%3A00Z&sig=QdHdwRc9z1YmndGapmguAEiPrJ3IDwMITLcD1gsN3Ug%3D&ske=2026-02-25T16%3A32%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-02-24T16%3A32%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=a38214881fc7108f387130af6ef79dca25ba6ef7838f47210e35e2e8fb711a95)

Caused by:
    HTTP status server error (503 Egress is over the account limit.) for url (https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:fdbe84136b3dd46d94305ef37f24f3cf04a70cc2026dca2592ac2ec0c9de15c7?se=2026-02-24T18%3A05%3A00Z&sig=QdHdwRc9z1YmndGapmguAEiPrJ3IDwMITLcD1gsN3Ug%3D&ske=2026-02-25T16%3A32%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-02-24T16%3A32%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=a38214881fc7108f387130af6ef79dca25ba6ef7838f47210e35e2e8fb711a95)

Some example logs are:

I don't know if this was an outage on GitHub's side, or what this is related to. I wanted to raise this here though since wkg's primary suorce is, I believe, ghcr artifacts for WASI. This is partly why I was curious about adding artifacts as in my experience github release artifacts have been pretty reliable (not perfect by any means...).

Regardless I wanted to raise this in case others had thoughts on this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the linked GitHub Actions job logs with the reported wkg OCI download failures, including the repeated invalid-token warnings and 503 egress-limit response. Determine whether the failure is a GitHub GHCR outage or a reproducible issue in the wkg download path; done means documenting a confirmed cause and a specific mitigation or follow-up.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, rust
Domain
ci-cd, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.