google / google/gvisor

nvproxy: `frontendFD`s are present during `runsc checkpoint` despite being closed

Open
#12,144 10 comments 0 reactions 0 assignees View on GitHub
stale-issue type: bug
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

### Description

Hello! I am currently using `cuda-checkpoint` within gVisor, and `runsc checkpoint` is failing with [nvproxy.frontendFD is not saveable](https://github.com/google/gvisor/blob/master/pkg/sentry/devices/nvproxy/save_restore_impl.go#L37). From my understanding, this error means that a `/dev/nvidia#`, `/dev/nvidiactl`, or `/dev/nvidia-uvm` file descriptor is open at the time of a gVisor snapshot. However, running `lsof -nP` inside the container after `cuda-checkpoint` indicates that none of these file descriptors are currently open. Why is there a discrepancy between `lsof` and the `nvproxy.frontendFDs` set? Is it possible that gVisor has a bug where it holds on to these `frontendFD` file descriptors even if they're closed by the guest?

### Steps to reproduce

I came across this issue when attempting to checkpoint/restore [sglang](https://github.com/sgl-project/sglang).

**Step 1: Build the image from this Dockerfile**
```
FROM nvidia/cuda:12.9.1-devel-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
wget \
git \
python3 \
python3-pip \
lsof \
&& rm -rf /var/lib/apt/lists/*

# Install CUDA Toolkit (nvcc)
RUN arch="x86_64" && \
distro="debian12" && \
filename="cuda-keyring_1.1-1_all.deb" && \
cuda_keyring_url="https://developer.download.nvidia.com/compute/cuda/repos/${distro}/${arch}/${filename}" && \
wget ${cuda_keyring_url} && \
dpkg -i ${filename} && \
rm -f ${filename} && \
apt-get update && \
apt-get install -y cuda-nvcc-12-9 && \
rm -rf /var/lib/apt/lists/*

RUN apt-get update -y && apt-get upgrade -y
RUN apt-get install numactl -y

# Clone and install sglang
RUN pip install --upgrade pip setuptools
RUN git clone https://github.com/sgl-project/sglang.git /sglang && \
cd /sglang && \
pip install -e "python[all]"

RUN wget https://github.com/NVIDIA/cuda-checkpoint/raw/refs/heads/main/bin/x86_64_Linux/cuda-checkpoint /usr/bin/cuda-checkpoint

CMD ["python3", "-c", "import sglang.srt.entrypoints.engine; import time; import os; print(f'pid: {os.getpid()}', flush=True); time.sleep(1000)"]
```

**Step 2: Prepare the container**
Follow the instructions [here](https://gvisor.dev/docs/user_guide/quick_start/oci/) to create an OCI bundle and config. Make make sure to setup [GPU support](https://gvisor.dev/docs/user_guide/gpu/). [This guide](https://gvisor.dev/blog/2023/06/20/gpu-pytorch-stable-diffusion/) is useful.

**Step 3: Running the container**
Run the container: `runsc -nvproxy run -bundle bundle/ sglang-container`

Then, once the PID is printed, run `cuda-checkpoint` inside the container: `runsc exec sglang-container cuda-checkpoint --toggle --pid 1`. This step should succeed.

You can now verify that no handles to `/dev/nvidia*` are present: `runsc exec sglang-container lsof -nP`.

**Step 4: Checkpoint the container**
This step fails: `checkpoint -image-path checkpointed_img/ sglang-container` with the error message pasted below. I've also uploaded the container's `lsof -nP` output, grepped for `cuda|nvidia`. I'm also seeing lots of the following, and I'm not sure if it's related.

```
object.go:260 [ 1689: 1689] nvproxy: freeing object with unknown handle 0xc1d09e26:0x5c000006
frontend.go:76 [ 1707: 1707] nvproxy: failed to open nvidia0: no such file or directory
```

**Log output**
* [runsc_boot_logs.txt](https://github.com/user-attachments/files/22394081/runsc_boot_logs.txt)
* [lsof_post_cuda_checkpoint.txt](https://github.com/user-attachments/files/22394000/lsof_post_cuda_checkpoint.txt)

### runsc version

```shell

```

### docker version (if using docker)

```shell

```

### uname

oracle linux 9.6, kernel 5.15.0-309.180.4.el9uek.x86_64

### kubectl (if using Kubernetes)

```shell
N/A
```

### repo state (if built from source)

_No response_

### runsc debug logs (if available)

```shell
See the above file upload.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.