Remove nvsnap-cuda-checkpoint CLI now that upstream ships arm64
@balajinvda is already working on this.
Since Aug 9, 2026.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
NVIDIA/cuda-checkpoint now publishes an aarch64 prebuilt (bin/aarch64_Linux), which removes the reason we built our own CLI.
Background
docker/agent/nvsnap-cuda-checkpoint.c is a drop-in replacement for the upstream cuda-checkpoint binary, built on the public driver checkpoint API (cuCheckpointProcess*). Dockerfile.base records two reasons for it:
buildable for x86-64 AND arm64 (NVIDIA publishes x86-64 only), and extensible (r580 gpuPairs migration, lock timeouts)
Reason 1 is now resolved upstream. Reason 2 is a judgement call (see below).
Cleanup scope
docker/agent/nvsnap-cuda-checkpoint.ccuda-cli-builderstage indocker/agent/Dockerfile.base- copy logic in
scripts/build-agent.sh(~lines 148-170) docker/agent/cuda-checkpoint-wrapper.sh(re-evaluate; it points LD_LIBRARY_PATH at the node driver libcuda)
Blockers to check before removing
-
Driver floor on arm. Upstream lists ARM CPU support as r595+. Our Dockerfile targets r570+ and the CLI header notes 550+ for basic actions. Switching to the upstream arm binary raises the arm driver requirement to r595. Inventory arm node driver versions first; if any sit below r595 this trades a working path for a broken one.
-
glibc compatibility. Commit 499628524 ("self-contained CRIU bundle via RPATH so cuda-checkpoint uses the container glibc") fixed a real bug in this area. Building from source let us control link-time glibc; a prebuilt has fixed expectations. Needs a smoke test against our actual workload images, since this class of failure passes unit tests and only aborts at restore.
-
Extensibility. Owning the source is the hook for
gpuPairsGPU migration (r580+), which is the direction multi-GPU work in #25 has been heading. Confirm the upstream binary exposes migration before giving up the hook.
Sequencing
Hold the removal until the in-flight manifest/chart cluster validation completes. Stacking a toolchain swap on top of unvalidated manifest changes makes any failure hard to attribute.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.