kvcache-ai / kvcache-ai/Mooncake
[CI] cuda-toolkit network install flakes on NVIDIA apt mirror sync (ubuntu2404 Packages size mismatch)
- Dominant language
- C++
- Stars
- 6.6k
- Forks
- 1.2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 312
Description
## Symptom
`test-wheel-ubuntu` (and other jobs using `Jimver/cuda-toolkit` with `method: network`) fail during **Install CUDA Toolkit** before any Mooncake tests run.
Seen on PR #3969 run https://github.com/kvcache-ai/Mooncake/actions/runs/34425688979/job/102715019979
## Log excerpt
```
E: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/Packages
File has unexpected size (1799720 != 1817210). Mirror sync in progress? [IP: …]
##[error]Error: The process '/usr/bin/sudo' failed with exit code 100
```
Also warns CUDA apt source is configured twice (`archive_uri-…noble.list` + `cuda-ubuntu2404-x86_64.list`).
## Context
- `ci.yml` pins `Jimver/cuda-toolkit@v0.2.24`
- `_build-efa-wheel.yaml` already uses `@v0.2.29`
- Failure is infra/mirror, not product code; CI Gate then fails because `test-wheel-ubuntu` failed
## Proposed fix (separate PR)
1. Retry the Install CUDA Toolkit step (2–3×) on apt size-mismatch / exit 100
2. Bump `Jimver/cuda-toolkit` in `ci.yml` toward the version already used by efa wheel (or newer), and avoid duplicate apt sources if still present
3. Optional: document that empty re-run is OK for this flake
Happy to send a small CI-only PR.
Contributor guide
Research direction
Start with ci.yml and the failing test-wheel-ubuntu job, then compare its Jimver/cuda-toolkit version with _build-efa-wheel.yaml. Reproduce or inspect the Install CUDA Toolkit step and its apt-source warnings. Done means the workflow tolerates the documented Packages size-mismatch failure, uses the chosen toolkit version, avoids duplicate sources where applicable, and the affected CI job passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, linux, ubuntu
- Domain
- ci-cd, devops
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100