agent-substrate / agent-substrate/substrate

Snapshot/restore for GPU actors (suspend/resume with a live CUDA context)

Offen
#779 3 Kommentare 0 Reaktionen 1 zugewiesene Person Beansprucht von @eliranw Auf GitHub ansehen
area/gvisor area/microVM kind/feature
Vorherrschende Sprache
Go
Sterne
1.8k
Forks
316
Ø Merge
2 T. 43 Min.
Gemergte PRs (30 T.)
287

Beschreibung

## Goal
Let a GPU actor suspend and resume while the workload is holding a CUDA context. Follow-up to the "Snapshot/restore with GPU out of scope" note in #627.

## What happens today (gVisor)
A Full-scope snapshot of an actor with a live CUDA context fails in runsc:

```
FATAL ERROR: checkpoint failed: checkpointing container "pause":
encoding error: can't save with live nvproxy clients
```

The sandbox dies with it, so retries get `cannot checkpoint container "pause" in state stopped`. The actor ends up in STATUS_SUSPENDING, which you can't delete from, holding its worker. Same as #50.

## Suggested approach (gVisor)
runsc already supports NVIDIA's cuda-checkpoint, which moves device memory to host and releases the GPU before saving. I tested it on a T4 with the runsc we pin (release-20260622.0): copied the binary into a running actor's rootfs and ran `runsc checkpoint -cuda-checkpoint-path /cuda-checkpoint` by hand. It went through, 301 MB of image files, no nvproxy error, workload still running after.

1. Get cuda-checkpoint into the actor. The flag wants a path inside the container, so it can ride the CDI mounts we already inject.
2. Pass `-cuda-checkpoint-path` in `cmdCheckpoint` (`cmd/ateom-gvisor/runsc.go`).
3. Restore needs nothing. gVisor replays the setting from the checkpoint image.

## Micro-VM
Micro-VM actors don't have GPU support yet (#762), but the snapshot story should be settled before that lands, because it may constrain the design.

Need more exploration in this area.

## Known costs / open issues
- **The binary isn't in the driver or the toolkit.** It's in-tree at `NVIDIA/cuda-checkpoint` under `bin/x86_64_Linux/`, no releases. About 6 KB, dynamically linked, so the actor image needs a loader. Same constraint that stopped distroless ateom from running nvidia-ctk.
- **Golden snapshots are always Full**, regardless of onPause/onCommit, since a data-only golden leaves nothing to restore the guest from. So a template holding a context at golden-snapshot time is affected either way.

Related: #627, #502, #762

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.