NVIDIA / NVIDIA/nvcf

Remove nvsnap-cuda-checkpoint CLI now that upstream ships arm64

Open
#730 1 comment 0 reactions 1 assignee View on GitHub

@balajinvda is already working on this.

Since Aug 9, 2026.

nvsnap
Dominant language
Go
Stars
218
Forks
72
Avg merge
1d 12h
Merged PRs (30d)
427

Description

NVIDIA/cuda-checkpoint now publishes an aarch64 prebuilt (bin/aarch64_Linux), which removes the reason we built our own CLI.

Background

docker/agent/nvsnap-cuda-checkpoint.c is a drop-in replacement for the upstream cuda-checkpoint binary, built on the public driver checkpoint API (cuCheckpointProcess*). Dockerfile.base records two reasons for it:

buildable for x86-64 AND arm64 (NVIDIA publishes x86-64 only), and extensible (r580 gpuPairs migration, lock timeouts)

Reason 1 is now resolved upstream. Reason 2 is a judgement call (see below).

Cleanup scope

  • docker/agent/nvsnap-cuda-checkpoint.c
  • cuda-cli-builder stage in docker/agent/Dockerfile.base
  • copy logic in scripts/build-agent.sh (~lines 148-170)
  • docker/agent/cuda-checkpoint-wrapper.sh (re-evaluate; it points LD_LIBRARY_PATH at the node driver libcuda)

Blockers to check before removing

  1. Driver floor on arm. Upstream lists ARM CPU support as r595+. Our Dockerfile targets r570+ and the CLI header notes 550+ for basic actions. Switching to the upstream arm binary raises the arm driver requirement to r595. Inventory arm node driver versions first; if any sit below r595 this trades a working path for a broken one.

  2. glibc compatibility. Commit 499628524 ("self-contained CRIU bundle via RPATH so cuda-checkpoint uses the container glibc") fixed a real bug in this area. Building from source let us control link-time glibc; a prebuilt has fixed expectations. Needs a smoke test against our actual workload images, since this class of failure passes unit tests and only aborts at restore.

  3. Extensibility. Owning the source is the hook for gpuPairs GPU migration (r580+), which is the direction multi-GPU work in #25 has been heading. Confirm the upstream binary exposes migration before giving up the hook.

Sequencing

Hold the removal until the in-flight manifest/chart cluster validation completes. Stacking a toolchain swap on top of unvalidated manifest changes makes any failure hard to attribute.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.