Repository navigation
Hosting WITs via OCI on GHCR is flaky #886
Description
Today we did a security release of Wasmtime which involves doing a lot of CI all at once. We had lots of flaky failures due to wkg being unable to download WITs, such as:
+ wkg get --format wit --overwrite wasi:cli@0.2.6 -o crates/wasi/src/p2/wit/deps/cli.wit
Getting wasi:cli@0.2.6...
2026-02-24T17:57:34.381595Z WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
2026-02-24T17:57:34.595113Z WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
2026-02-24T17:57:34.946847Z WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
2026-02-24T17:57:36.054084Z WARN oci_client::token_cache: Invalid bearer token error=Error(InvalidToken)
Error: registry error: HTTP status server error (503 Egress is over the account limit.) for url (https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:fdbe84136b3dd46d94305ef37f24f3cf04a70cc2026dca2592ac2ec0c9de15c7?se=2026-02-24T18%3A05%3A00Z&sig=QdHdwRc9z1YmndGapmguAEiPrJ3IDwMITLcD1gsN3Ug%3D&ske=2026-02-25T16%3A32%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-02-24T16%3A32%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=a38214881fc7108f387130af6ef79dca25ba6ef7838f47210e35e2e8fb711a95)
Caused by:
HTTP status server error (503 Egress is over the account limit.) for url (https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:fdbe84136b3dd46d94305ef37f24f3cf04a70cc2026dca2592ac2ec0c9de15c7?se=2026-02-24T18%3A05%3A00Z&sig=QdHdwRc9z1YmndGapmguAEiPrJ3IDwMITLcD1gsN3Ug%3D&ske=2026-02-25T16%3A32%3A44Z&skoid=fb3d2a07-ec6c-4fe4-aced-9efe0fd2fe1a&sks=b&skt=2026-02-24T16%3A32%3A44Z&sktid=398a6654-997b-47e9-b12b-9515b896b4de&skv=2025-01-05&sp=r&spr=https&sr=b&sv=2025-01-05&hmac=a38214881fc7108f387130af6ef79dca25ba6ef7838f47210e35e2e8fb711a95)
Some example logs are:
- https://github.com/bytecodealliance/wasmtime/actions/runs/22363293742/job/64722423879
- https://github.com/bytecodealliance/wasmtime/actions/runs/22363788079/job/64724254926
- https://github.com/bytecodealliance/wasmtime/actions/runs/22363534893/job/64723336690
- https://github.com/bytecodealliance/wasmtime/actions/runs/22361528404/job/64716132428
- https://github.com/bytecodealliance/wasmtime/actions/runs/22361528404/job/64722334417
- https://github.com/bytecodealliance/wasmtime/actions/runs/22361528404/job/64722932225
I don't know if this was an outage on GitHub's side, or what this is related to. I wanted to raise this here though since wkg's primary suorce is, I believe, ghcr artifacts for WASI. This is partly why I was curious about adding artifacts as in my experience github release artifacts have been pretty reliable (not perfect by any means...).
Regardless I wanted to raise this in case others had thoughts on this.
Activity
ricochet commentedon Feb 24, 2026
This is a related to but different from the error that was reported in #873
I think this problem is multi-fold:
alexcrichton commentedon Feb 24, 2026
If this ends up mostly being an issue in
wkgI think it's reasonable to chalk it up to that and stiack with ghcr for now, I mostly just wasn't sure if this was awkgthing or a GitHub thing.