microsoft / microsoft/WSL

[BUG] CUDA Unified Memory (nvbuf-memory-type=3) CPU-mapping segfaults in libpixman under GPU-PV on Blackwell (RTX 5070) -- reproduces on driver 596.49 AND 595.97

Open
#41,483 9 comments 0 reactions 0 assignees View on GitHub
emailed-logs GPU wsl2
Dominant language
C++
Stars
33.7k
Forks
1.8k
Avg merge
3d 17h
Merged PRs (30d)
116

Description

## Summary

A GStreamer/DeepStream pipeline running under Docker Desktop + WSL2 GPU passthrough on an RTX 5070 (Blackwell, sm_120) reliably segfaults within 10-90 seconds when a CUDA buffer allocated as **CUDA Unified/Managed memory** is mapped for CPU read/write while the GPU is concurrently active on other buffers. The crash reproduces **identically on two separate NVIDIA driver versions** (596.49 and 595.97 / R595), always in `libpixman-1.so` inside the process touching the mapped memory, with the same signature both times. Switching the same buffer to plain device (VRAM-resident, non-unified) memory and doing the rare CPU-side touch via an explicit `cuMemcpyDtoH`/`cuMemcpyHtoD` round-trip through a pinned staging buffer (never mapping the unified pointer directly) makes the crash disappear entirely — confirmed stable across multiple independent runs of several minutes each.

This looks like the general "CPU must not access CUDA managed memory while the GPU is actively executing" constraint being violated in a way that native Linux tolerates (or handles more gracefully) but WSL2's GPU-PV (paravirtualized GPU passthrough) layer turns into a hard segfault on Blackwell.

## Environment

- GPU: NVIDIA GeForce RTX 5070 (Blackwell, sm_120), 12 GB VRAM
- CPU: Intel i9-12900K (24 threads)
- Host OS: Windows 11 Pro
- WSL2 kernel: `6.18.40.1-microsoft-standard-WSL2`
- Docker: Docker Desktop, engine 29.7.2, WSL2-based engine (not Hyper-V backend)
- NVIDIA drivers tested (both reproduce identically): `596.49` and `595.97` (R595)
- CUDA: 13.2 (per `nvidia-smi`)
- Application: NVIDIA DeepStream 9.0 (Triton), a GStreamer pipeline with 5 concurrent camera sources, each with its own decode/infer/OSD chain
- The affected buffer: the RGBA OSD compositing buffer between `nvdsosd` and the downstream video converter, allocated via `nvvideoconvert`'s `nvbuf-memory-type` property

## Reproduction pattern

1. Allocate a GStreamer NVMM buffer with `nvbuf-memory-type=3` (`NVBUF_MEM_CUDA_UNIFIED`).
2. On a low-frequency callback (a GStreamer pad probe firing only on frames that need an extra CPU-side overlay draw — not every frame), call `NvBufSurfaceMap(surf, -1, -1, NVBUF_MAP_READ_WRITE)` to get a CPU-writable pointer into that same unified-memory buffer, do a small amount of CPU-side drawing into it (Cairo, via `libpixman`), then `NvBufSurfaceSyncForDevice` + `NvBufSurfaceUnMap`.
3. Meanwhle, every other frame (no CPU touch), the GPU (`nvdsosd`, the converter, the encoder) is continuously reading/writing that same buffer pool.
4. Within 10-90 seconds of running 5 concurrent camera pipelines like this, the process segfaults. Confirmed via `dmesg`:

```
[ 5650.232899] q-osd:src[2541]: segfault at 23ea78794 ip 000075d887bd807f sp 000075d254fb6e38 error 4 in libpixman-1.so.0.42.2[7207f,75d887b70000+8d000] likely on CPU 20 (core 10, socket 0)
```

and, on a separate test run after downgrading the driver from 596.49 to 595.97 (R595), the same signature at a different (faster) time:

```
[ 5977.388037] q-osd:src[5623]: segfault at 23f1fade4 ip 00007f9d7677e07f sp 00007f96e2fbae38 error 4 in libpixman-1.so.0.42.2[7207f,7f9d76716000+8d000] likely on CPU 7 (core 3, socket 0)
```

The thread name (`q-osd:src`) is the GStreamer OSD queue thread; every crash is inside `libpixman-1.so`'s pixel-copy routines, i.e. literally while the CPU is reading/writing pixels through the mapped pointer.

## What we ruled out before concluding this is memory-type-related

- **Not a Cairo/font thread-safety bug**: added a `std::mutex` serializing all Cairo drawing calls across the 5 per-camera GStreamer threads first (since `cairo_select_font_face`/`cairo_show_text` share a process-wide FreeType cache). The mutex is legitimate and stays, but it did **not** fix this crash — it only fixed a separate, unrelated flicker issue.
- **Not driver-version-specific**: reproduces identically on 596.49 and on 595.97 (R595). If anything, R595 crashed *faster* (10s vs. 45s).
- **Not a newer-WSL2-fixes-it situation**: updated WSL2 from 2.7.12.0 → 2.9.9.0 (kernel 6.18.33.2-2 → 6.18.40.1-1) via `wsl --update --pre-release`. Identical crash signature persisted on the new kernel.
- **Not fixable by switching Docker Desktop's backend**: tried the Hyper-V backend instead of WSL2 — that backend doesn't support NVIDIA container GPU passthrough at all (`nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1`), so it's not a viable alternative, and doesn't tell us anything about the underlying GPU-PV layer either way.
- **Confirmed fixed by avoiding the CPU-mapped-unified-memory pattern entirely**: switched the same buffer to `nvbuf-memory-type=2` (`NVBUF_MEM_CUDA_DEVICE`, plain VRAM-resident, not CPU-mappable) and replaced the `NvBufSurfaceMap`/`SyncForCpu`/`SyncForDevice` CPU-touch with an explicit CUDA driver-API copy: `cuMemcpyDtoH` into a reusable pinned (`cuMemAllocHost`) host staging buffer, draw with Cairo on that host buffer, `cuMemcpyHtoD` back — done only for the rare frame that needs the extra draw, so the buffer stays GPU-resident on every other frame, but the CPU never touches the actual device/unified pointer. This has run cleanly for 8+ minutes and separately for 6+ minutes in independent test runs (vs. the 10-90s crash window with the mapped-unified-memory approach), with normal GPU utilization and VRAM usage.
- **Confirmed side-effect of type=3, independent of the crash**: while `nvbuf-memory-type=3` was active (even briefly, before it crashed), `nvidia-smi` showed the pipeline using only ~900 MB of GPU memory at ~24% utilization (vs. ~3.4-4.2 GB / 45-54% with type=2), and `docker stats` showed elevated container CPU usage — consistent with buffers silently migrating to host RAM instead of staying GPU-resident, on top of the eventual crash.

## Why this looks like a WSL2 GPU-PV / Unified Memory platform issue, not an app bug

NVIDIA's own [CUDA on WSL User Guide](https://docs.nvidia.com/cuda/wsl-user-guide/index.html) documents that Unified/Managed memory under WSL2 (like native Windows) does not support the same oversubscription / page-fault-migration model as native Linux — managed memory is first allocated in CPU physical memory and migrated in different granularity. That's a documented *behavioral* difference; it's not clear from the docs whether "CPU touches unified memory concurrently with active GPU use on the same buffer pool, on Blackwell, under GPU-PV" is expected to be safe or is a known-unsafe pattern that should be documented as unsafe. Given it reproduces identically across two driver versions and a WSL2 kernel update, my working theory is this is a GPU-PV (`dxgkrnl`) + Blackwell interaction, not something an NVIDIA driver point-release will fix.

## Related but distinct issues found while researching this

- [NVIDIA/cuda-samples#433](https://github.com/NVIDIA/cuda-samples/issues/433) — `cuInit`-time SIGSEGV on driver 596.49/Blackwell caused by a `libcudart` 13.2.1 vs. driver mismatch. Different trigger point (init-time vs. ours, which happens well after CUDA/the pipeline are running normally) and a different fix (downgrading `libcudart`) that doesn't apply to what we're seeing.
- [microsoft/WSL#40401](https://github.com/microsoft/WSL/issues/40401) — 16 GiB CUDA driver context overhead on Blackwell sm_120 under WSL2 GPU passthrough.
- [microsoft/WSL#40333](https://github.com/microsoft/WSL/issues/40333) — Blackwell FP8 tensor cores not exposed via `dxgkrnl` in WSL2.
- [microsoft/WSL#40732](https://github.com/microsoft/WSL/issues/40732) — WSL2 GPU OOM leading to Hyper-V kernel panic / BSOD (different trigger, same general "GPU-PV state gets corrupted under memory pressure on Blackwell" theme).
- [microsoft/WSL#41361](https://github.com/Microsoft/wsl/issues/41361) — llama.cpp CUDA hangs non-deterministically on RTX 5070 Ti (Blackwell) under WSL2.

## Questions for the WSL/NVIDIA teams

1. Is "CPU maps/touches a `NVBUF_MEM_CUDA_UNIFIED` (or plain `cudaMallocManaged`) buffer while the GPU is concurrently active on other regions of the same buffer pool" a known-unsafe pattern specifically under WSL2 GPU-PV on Blackwell? If so, is that documented anywhere users would find it before hitting a segfault?
2. Is there a recommended, supported way to safely CPU-touch a small subset of a GPU-active buffer pool under WSL2 GPU-PV on Blackwell, other than avoiding Unified Memory entirely and doing an explicit pinned-buffer copy (which is what we ended up doing)?
3. Is this specific interaction (Unified Memory + concurrent GPU use + Blackwell + GPU-PV) on any internal tracking/roadmap, or should this be considered a permanent platform limitation?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reported WSL2 GPU-PV failure with the DeepStream/GStreamer pipeline, RTX 5070, and nvbuf-memory-type=3 pattern described here. Compare the mapped unified-memory path with the explicit pinned-buffer copy path and review the linked WSL and CUDA issues. Done means establishing whether this is a supported limitation or a platform defect, with the responsible component and any documentation or fix requirement identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux
Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.