LizardByte / LizardByte/Sunshine

Linux NVENC fails when CUDA device has no matching DRM node (/dev/dxg / Hyper-V GPU-P)

Open
#5,598 1 comment 0 reactions 0 assignees View on GitHub
ai
Dominant language
C++
Stars
41.3k
Forks
2.1k
Avg merge
23h 47m
Merged PRs (30d)
124

Description

## Description

Sunshine cannot use NVENC on Linux when the CUDA/NVENC device is available but has no matching Linux DRM node.

This is reproducible with an NVIDIA GPU exposed to a Linux Hyper-V guest through GPU-P. The GPU is fully usable through `/dev/dxg`: NVML, CUDA, NVDEC, and direct NVENC all work, but Sunshine fails while trying to map the CUDA device to a DRM device.

The underlying limitation appears generic rather than Hyper-V-specific: Sunshine assumes that a CUDA device used for NVENC also has a corresponding DRM node under `/sys/bus/pci/devices//drm`.

## Environment

- Host: Windows 11, Hyper-V
- Guest: Gen2 Linux VM, Omarchy / Arch Linux
- Kernel: `7.1.9.arch1-2`
- GPU: NVIDIA GeForce RTX 4050 via GPU-P
- NVIDIA host driver: `596.08`
- GPU interface in guest: `/dev/dxg`
- Display device in guest: `hyperv_drm`
- `/dev/dri`: only `card1` from `hyperv_drm`; no NVIDIA render node
- Sunshine installed version: `2026.516.143833-1`

Also tested unchanged official Sunshine AppImages:
- `2025.924.154138` (`86188d47`)
- `2026.516.143833` (`14ffa6fd`)
- `2026.902.143621` (`b91470ff`)

All show the same behavior.

## GPU functionality outside Sunshine

The GPU-P path itself works correctly:

- `/dev/dxg`: PASS
- NVML / `nvidia-smi`: PASS
- CUDA: PASS
- NVDEC: PASS
- Direct NVENC API: PASS
- Direct NVENC stress test: 12,000 frames encoded successfully at 1080p, approximately 251-349 fps

So the NVIDIA hardware, driver bridge, and NVENC API are functional in the guest.

## Reproduction

Configure Sunshine with NVENC and use either KMS or wlroots capture.

Example relevant setting:

```ini
encoder = nvenc
```

With KMS capture, Sunshine reaches the CUDA/NVENC path but fails with:

```text
Unable to find DRM device with PCI bus ID: 0000:01:00.0
```

The same failure occurs with wlroots capture.

CUDA reports `0000:01:00.0`, but there is no corresponding NVIDIA DRM device in the guest. GPU-P exposes the GPU through `/dev/dxg`, not through a normal Linux DRM render node.

The only DRM card is the Hyper-V display device:

```text
/dev/dri/card1 -> hyperv_drm
```

There is no matching:

```text
/sys/bus/pci/devices/0000:01:00.0/drm
/dev/dri/renderD*
```

## Observed behavior

Sunshine calls the DRM lookup for the CUDA device and aborts NVENC initialization when no matching DRM node exists.

This happens even though direct NVENC encoding on the same CUDA device works correctly.

Both KMS and wlroots capture hit the same limitation.

## Expected behavior

If CUDA/NVENC is available but the CUDA device has no matching DRM node, Sunshine should ideally be able to fall back to a system-memory capture/upload path instead of rejecting the encoder.

Conceptually:

```text
capture
-> system memory
-> CUDA upload
-> NVENC
```

This would avoid requiring DRM/DMABUF interop for devices that expose CUDA/NVENC through a non-DRM interface such as `/dev/dxg`.

## Code-path observations

There already appears to be a DRM-free CUDA RAM path through `cuda_ram_t` / `cuda::make_avcodec_encode_device(..., false)`.

The current failure appears to happen later when the encode device is initialized:

- KMS can fall back from `display_vram_t` to `display_ram_t`, but NVENC initialization subsequently attempts the CUDA-to-DRM lookup.
- wlroots currently selects a VRAM-oriented CUDA path and does not appear to fall back to a RAM capture path for this case.
- `open_drm_fd_for_cuda_device()` requires a matching DRM node and therefore cannot succeed with `/dev/dxg`.

PR #4946 may be relevant because it introduces a SHM/RAM capture fallback and `wl::force_ram_capture`. I am not claiming that it fixes this case as-is, but its RAM-capture routing looks conceptually aligned with what is needed here.

## Not a regression between recent releases

I tested the same setup against Sunshine versions spanning roughly a year:

| Version | Capture | Result |
|---|---|---|
| 2025.924.154138 | KMS | fails at CUDA -> DRM lookup |
| 2026.516.143833 | KMS | same failure |
| 2026.902.143621 | KMS | same failure |
| 2026.902.143621 | wlroots | same failure |

The relevant CUDA-to-DRM assumption therefore appears older than these releases rather than being a recent regression.

## Why this matters beyond this exact setup

GPU-P is only one way to reach this state. The more general case is:

> A valid CUDA/NVENC device exists, but there is no corresponding Linux DRM node.

Sunshine currently cannot use NVENC in that situation even when the encoder itself works.

A generic RAM-capture/CUDA-upload fallback would make the NVENC path more resilient and avoid coupling encoder availability to the presence of a DRM peer when zero-copy DRM interop is unavailable.

I can provide additional logs or test patches if useful.

Contributor guide

Open the contributing guide

Research direction

Start by tracing cuda_ram_t and cuda::make_avcodec_encode_device(..., false), then inspect open_drm_fd_for_cuda_device() and the KMS and wlroots capture paths. Compare the RAM-capture routing and wl::force_ram_capture changes in PR #4946. Done means NVENC initializes through a system-memory capture and CUDA upload path when no matching DRM node exists, while existing DRM-backed paths continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
computer-graphics, operating-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.