agent-substrate / agent-substrate/substrate

Snapshot/restore for GPU actors (suspend/resume with a live CUDA context)

Abierto
#779 3 comentarios 0 reacciones 1 asignado Reclamado por @eliranw Ver en GitHub
area/gvisor area/microVM kind/feature
Lenguaje dominante
Go
Estrellas
1.8k
Forks
316
Merge medio
2 d 43 min
PR fusionados (30 d)
287

Descripción

## Goal
Let a GPU actor suspend and resume while the workload is holding a CUDA context. Follow-up to the "Snapshot/restore with GPU out of scope" note in #627.

## What happens today (gVisor)
A Full-scope snapshot of an actor with a live CUDA context fails in runsc:

```
FATAL ERROR: checkpoint failed: checkpointing container "pause":
encoding error: can't save with live nvproxy clients
```

The sandbox dies with it, so retries get `cannot checkpoint container "pause" in state stopped`. The actor ends up in STATUS_SUSPENDING, which you can't delete from, holding its worker. Same as #50.

## Suggested approach (gVisor)
runsc already supports NVIDIA's cuda-checkpoint, which moves device memory to host and releases the GPU before saving. I tested it on a T4 with the runsc we pin (release-20260622.0): copied the binary into a running actor's rootfs and ran `runsc checkpoint -cuda-checkpoint-path /cuda-checkpoint` by hand. It went through, 301 MB of image files, no nvproxy error, workload still running after.

1. Get cuda-checkpoint into the actor. The flag wants a path inside the container, so it can ride the CDI mounts we already inject.
2. Pass `-cuda-checkpoint-path` in `cmdCheckpoint` (`cmd/ateom-gvisor/runsc.go`).
3. Restore needs nothing. gVisor replays the setting from the checkpoint image.

## Micro-VM
Micro-VM actors don't have GPU support yet (#762), but the snapshot story should be settled before that lands, because it may constrain the design.

Need more exploration in this area.

## Known costs / open issues
- **The binary isn't in the driver or the toolkit.** It's in-tree at `NVIDIA/cuda-checkpoint` under `bin/x86_64_Linux/`, no releases. About 6 KB, dynamically linked, so the actor image needs a loader. Same constraint that stopped distroless ateom from running nvidia-ctk.
- **Golden snapshots are always Full**, regardless of onPause/onCommit, since a data-only golden leaves nothing to restore the guest from. So a template holding a context at golden-snapshot time is affected either way.

Related: #627, #502, #762

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.