kvcache-ai / kvcache-ai/Mooncake

[CI] cuda-toolkit network install flakes on NVIDIA apt mirror sync (ubuntu2404 Packages size mismatch)

Open
#3,989 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
6.6k
Forks
1.2k
Avg merge
3d 5h
Merged PRs (30d)
312

Description

## Symptom
`test-wheel-ubuntu` (and other jobs using `Jimver/cuda-toolkit` with `method: network`) fail during **Install CUDA Toolkit** before any Mooncake tests run.

Seen on PR #3969 run https://github.com/kvcache-ai/Mooncake/actions/runs/34425688979/job/102715019979

## Log excerpt
```
E: Failed to fetch http://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/Packages
File has unexpected size (1799720 != 1817210). Mirror sync in progress? [IP: …]
##[error]Error: The process '/usr/bin/sudo' failed with exit code 100
```
Also warns CUDA apt source is configured twice (`archive_uri-…noble.list` + `cuda-ubuntu2404-x86_64.list`).

## Context
- `ci.yml` pins `Jimver/cuda-toolkit@v0.2.24`
- `_build-efa-wheel.yaml` already uses `@v0.2.29`
- Failure is infra/mirror, not product code; CI Gate then fails because `test-wheel-ubuntu` failed

## Proposed fix (separate PR)
1. Retry the Install CUDA Toolkit step (2–3×) on apt size-mismatch / exit 100
2. Bump `Jimver/cuda-toolkit` in `ci.yml` toward the version already used by efa wheel (or newer), and avoid duplicate apt sources if still present
3. Optional: document that empty re-run is OK for this flake

Happy to send a small CI-only PR.

Contributor guide

Open the contributing guide

Research direction

Start with ci.yml and the failing test-wheel-ubuntu job, then compare its Jimver/cuda-toolkit version with _build-efa-wheel.yaml. Reproduce or inspect the Install CUDA Toolkit step and its apt-source warnings. Done means the workflow tolerates the documented Packages size-mismatch failure, uses the chosen toolkit version, avoids duplicate sources where applicable, and the affected CI job passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, linux, ubuntu
Domain
ci-cd, devops
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.