NVFBC dual-monitor capture returns wrong output's content despite correct output_name/dwOutputId (raw NVFBC call proves driver-level capture is correct)

Is there an existing issue for this?

• I have searched the existing issues

Is your issue described in the documentation?

• I have read the documentation

Is your issue present in the latest beta/pre-release?

This issue is present in the latest pre-release (v2026.516.143833, matches latest at time of filing).

Describe the Bug

Running Sunshine with capture = nvfbc on a host with two NVIDIA outputs active simultaneously on one X screen (a "dual-head" Xorg config, not a typical single-monitor or mirrored setup — two distinct outputs, one Xorg Screen spanning both via MetaModes, e.g. DFP-0:1920x1080+0+0, DFP-1:2560x1440+1920+0).

With output_name set to the second output's index (1, correctly resolved by Sunshine itself as DP-0), the actual stream shown to the client is not DP-0's content — it is a 1:1 copy of the first output's content (output_name=0, HDMI-0), including a visibly mirrored/offset mouse cursor.

This is not a config-selection mistake: Sunshine's own log shows correct output enumeration and a correct dwOutputId/trackedBox for the selected output at every capture setup, both at startup and at stream-session start (see log excerpt below).

To isolate whether this was a driver/NVFBC bug, I wrote a minimal standalone C program that dlopen()s libnvidia-fbc.so.1 directly and drives the NvFBC API the same way cuda.cpp's handle_t/display_t does (including the privateData unlock blob for consumer GPUs), using the exact same dwOutputId Sunshine itself resolves and logs for the target output. That standalone call correctly captures the target output's content — verified both by inspecting raw pixel samples and by converting the raw BGRA buffer to a PNG and viewing it. No trace of the other output's content appeared anywhere in the frame.

This rules out the NVIDIA driver/NVFBC itself as the source of the wrong content. The discrepancy must be somewhere in Sunshine's own pipeline, between a successful NvFBCToCudaGrabFrame (or NvFBCToSysGrabFrame) call and the frame actually reaching the encoder/client.

Full source of the standalone test program (compiles standalone against the NvFBC.h vendored in this repo at third-party/nvfbc/NvFBC.h, no other dependency besides -ldl):

/* Standalone raw NVFBC test, bypassing Sunshine entirely. */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <dlfcn.h>
#include "NvFBC.h"

typedef NVFBCSTATUS(NVFBCAPI *PNVFBCCREATEINSTANCE)(NVFBC_API_FUNCTION_LIST *);

static void die(const char *what, NVFBC_API_FUNCTION_LIST *f, NVFBC_SESSION_HANDLE h) {
    fprintf(stderr, "FAILED: %s", what);
    if (f && f->nvFBCGetLastErrorStr && h) {
        fprintf(stderr, " -- %s", f->nvFBCGetLastErrorStr(h));
    }
    fprintf(stderr, "\n");
    exit(1);
}

int main(void) {
    void *lib = dlopen("libnvidia-fbc.so.1", RTLD_NOW);
    if (!lib) {
        fprintf(stderr, "dlopen failed: %s\n", dlerror());
        return 1;
    }

    PNVFBCCREATEINSTANCE NvFBCCreateInstance =
        (PNVFBCCREATEINSTANCE) dlsym(lib, "NvFBCCreateInstance");
    if (!NvFBCCreateInstance) {
        fprintf(stderr, "dlsym failed: %s\n", dlerror());
        return 1;
    }

    NVFBC_API_FUNCTION_LIST func;
    memset(&func, 0, sizeof(func));
    func.dwVersion = NVFBC_VERSION;

    if (NvFBCCreateInstance(&func) != NVFBC_SUCCESS) {
        fprintf(stderr, "NvFBCCreateInstance failed\n");
        return 1;
    }

    /* Consumer GeForce GPUs reject NvFBCCreateHandle() unless this exact
     * privateData blob is supplied. Taken verbatim from Sunshine's own
     * cuda.cpp (src/platform/linux/cuda.cpp, handle_t::make()), which in
     * turn credits keylase/nvidia-patch's nvfbcwrp_main.cpp. */
    static const unsigned int MAGIC_PRIVATE_DATA[4] = {
        0xAEF57AC5, 0x401D1A39, 0x1B856BBE, 0x9ED0CEBA
    };

    NVFBC_CREATE_HANDLE_PARAMS create_params;
    memset(&create_params, 0, sizeof(create_params));
    create_params.dwVersion = NVFBC_CREATE_HANDLE_PARAMS_VER;
    create_params.privateData = MAGIC_PRIVATE_DATA;
    create_params.privateDataSize = sizeof(MAGIC_PRIVATE_DATA);

    NVFBC_SESSION_HANDLE handle = 0;
    if (func.nvFBCCreateHandle(&handle, &create_params) != NVFBC_SUCCESS) {
        die("nvFBCCreateHandle", &func, 0);
    }

    NVFBC_BIND_CONTEXT_PARAMS bind_params;
    memset(&bind_params, 0, sizeof(bind_params));
    bind_params.dwVersion = NVFBC_BIND_CONTEXT_PARAMS_VER;
    if (func.nvFBCBindContext(handle, &bind_params) != NVFBC_SUCCESS) {
        die("nvFBCBindContext", &func, handle);
    }

    NVFBC_GET_STATUS_PARAMS status_params;
    memset(&status_params, 0, sizeof(status_params));
    status_params.dwVersion = NVFBC_GET_STATUS_PARAMS_VER;
    if (func.nvFBCGetStatus(handle, &status_params) != NVFBC_SUCCESS) {
        die("nvFBCGetStatus", &func, handle);
    }

    printf("=== Raw NVFBC enumeration (bypassing Sunshine) ===\n");
    printf("dwOutputNum: %u\n", status_params.dwOutputNum);
    printf("Virtual Desktop: %ux%u\n", status_params.screenSize.w, status_params.screenSize.h);
    printf("XRandR available: %s\n", status_params.bXRandRAvailable ? "yes" : "no");

    uint32_t dp0_id = 0;
    int dp0_found = 0;

    for (uint32_t i = 0; i < status_params.dwOutputNum; ++i) {
        NVFBC_RANDR_OUTPUT_INFO *o = &status_params.outputs[i];
        printf("-- Output index %u --\n", i);
        printf("  dwId: %u\n", o->dwId);
        printf("  name: %s\n", o->name);
        printf("  trackedBox: %ux%u @ %u,%u\n",
               o->trackedBox.w, o->trackedBox.h, o->trackedBox.x, o->trackedBox.y);

        if (o->trackedBox.w == 2560 && o->trackedBox.h == 1440 &&
            o->trackedBox.x == 1920 && o->trackedBox.y == 0) {
            dp0_id = o->dwId;
            dp0_found = 1;
        }
    }

    if (!dp0_found) {
        fprintf(stderr, "Could not identify DP-0 (2560x1440 @ 1920,0) in the enumeration above.\n");
        return 1;
    }

    printf("\nUsing dwOutputId=%u for DP-0 (matched by trackedBox geometry)\n\n", dp0_id);

    NVFBC_CREATE_CAPTURE_SESSION_PARAMS session_params;
    memset(&session_params, 0, sizeof(session_params));
    session_params.dwVersion = NVFBC_CREATE_CAPTURE_SESSION_PARAMS_VER;
    session_params.eCaptureType = NVFBC_CAPTURE_TO_SYS;
    session_params.eTrackingType = NVFBC_TRACKING_OUTPUT;
    session_params.dwOutputId = dp0_id;
    session_params.bWithCursor = NVFBC_FALSE;

    if (func.nvFBCCreateCaptureSession(handle, &session_params) != NVFBC_SUCCESS) {
        die("nvFBCCreateCaptureSession", &func, handle);
    }

    void *buffer = NULL;
    NVFBC_TOSYS_SETUP_PARAMS setup_params;
    memset(&setup_params, 0, sizeof(setup_params));
    setup_params.dwVersion = NVFBC_TOSYS_SETUP_PARAMS_VER;
    setup_params.eBufferFormat = NVFBC_BUFFER_FORMAT_BGRA;
    setup_params.ppBuffer = &buffer;

    if (func.nvFBCToSysSetUp(handle, &setup_params) != NVFBC_SUCCESS) {
        die("nvFBCToSysSetUp", &func, handle);
    }

    NVFBC_FRAME_GRAB_INFO grab_info;
    memset(&grab_info, 0, sizeof(grab_info));

    NVFBC_TOSYS_GRAB_FRAME_PARAMS grab_params;
    memset(&grab_params, 0, sizeof(grab_params));
    grab_params.dwVersion = NVFBC_TOSYS_GRAB_FRAME_PARAMS_VER;
    grab_params.dwFlags = NVFBC_TOSYS_GRAB_FLAGS_NOFLAGS;
    grab_params.pFrameGrabInfo = &grab_info;
    grab_params.dwTimeoutMs = 2000;

    if (func.nvFBCToSysGrabFrame(handle, &grab_params) != NVFBC_SUCCESS) {
        die("nvFBCToSysGrabFrame", &func, handle);
    }

    printf("=== Grabbed frame ===\n");
    printf("Width: %u\n", grab_info.dwWidth);
    printf("Height: %u\n", grab_info.dwHeight);
    printf("ByteSize: %u\n", grab_info.dwByteSize);

    FILE *out = fopen("/tmp/nvfbc_grab.bgra", "wb");
    if (!out) {
        fprintf(stderr, "Could not open output file\n");
        return 1;
    }
    fwrite(buffer, 1, grab_info.dwByteSize, out);
    fclose(out);

    printf("Wrote raw BGRA buffer to nvfbc_grab.bgra (%ux%u)\n", grab_info.dwWidth, grab_info.dwHeight);

    NVFBC_RELEASE_CONTEXT_PARAMS release_params;
    memset(&release_params, 0, sizeof(release_params));
    release_params.dwVersion = NVFBC_RELEASE_CONTEXT_PARAMS_VER;
    func.nvFBCReleaseContext(handle, &release_params);

    NVFBC_DESTROY_HANDLE_PARAMS destroy_params;
    memset(&destroy_params, 0, sizeof(destroy_params));
    destroy_params.dwVersion = NVFBC_DESTROY_HANDLE_PARAMS_VER;
    func.nvFBCDestroyHandle(handle, &destroy_params);

    return 0;
}

Compiled with: gcc -o nvfbc_test nvfbc_test.c -ldl (no CUDA, no X11 dev headers needed). Output when run against the dual-head session (DISPLAY=:99 ./nvfbc_test):

=== Raw NVFBC enumeration (bypassing Sunshine) ===
dwOutputNum: 2
Virtual Desktop: 4480x1440
XRandR available: yes
-- Output index 0 --
  dwId: 444
  name: HDMI-0
  trackedBox: 1920x1080 @ 0,0
-- Output index 1 --
  dwId: 460
  name: DP-0
  trackedBox: 2560x1440 @ 1920,0

Using dwOutputId=460 for DP-0 (matched by trackedBox geometry)

=== Grabbed frame ===
Width: 2560
Height: 1440
ByteSize: 14745600
Wrote raw BGRA buffer to nvfbc_grab.bgra (2560x1440)

The grabbed frame's dimensions (2560x1440) and byte size (exactly 2560*1440*4) are correct for DP-0, and the pixel content was verified to actually be DP-0's content (not HDMI-0's) both by sampling raw pixel values and by visual inspection after converting to PNG.

Expected Behavior

The stream should show the content of the output selected via output_name (DP-0 in this case, dwOutputId=460), not the first-enumerated output's content (HDMI-0, dwOutputId=444).

Additional Context

• Sunshine runs inside a custom Ubuntu 24.04 Docker container (runtime: nvidia, NVIDIA_DRIVER_CAPABILITIES=all, cap_add: SYS_ADMIN, SYS_NICE), installed from the official .deb release asset — not the official Docker image.
• Relevant sunshine.conf excerpt:

  capture = nvfbc
  encoder = nvenc
  adapter_name = /dev/dri/renderD128
  output_name = 1
  min_log_level = 0
  

• Xorg dual-head config (relevant Device/Screen sections):

  Option "ConnectedMonitor" "DFP-0,DFP-1"
  Option "CustomEDID" "DFP-0:<path>;DFP-1:<path>"
  Option "MetaModes" "DFP-0:1920x1080+0+0, DFP-1:2560x1440+1920+0"
  ...
  Virtual 4480 1440
  

• Happy to provide anything else useful on request (further log excerpts, the raw/PNG captured frames from the standalone test, etc.) — this is a private homelab setup so I've inlined what I can rather than linking out to a private repo.

Host Operating System

Linux

Operating System Version

Ubuntu 24.04.4 LTS (in Docker container; container host is Debian 13)

Architecture

amd64/x86_64

Package

Linux - deb

GPU Type

NVIDIA

GPU Model

GeForce RTX 4090

GPU Driver/Mesa Version

610.43.02

Capture Method

NvFBC (Linux)

Log output

[2026-08-20 08:57:15.056]: Info: Found [2] outputs
[2026-08-20 08:57:15.056]: Info: Virtual Desktop: 4480x1440
[2026-08-20 08:57:15.056]: Info: XrandR: available
[2026-08-20 08:57:15.056]: Info: -- Output --
[2026-08-20 08:57:15.056]: Info:   Resolution: 1920x1080
[2026-08-20 08:57:15.056]: Info:   Offset: 0x0
[2026-08-20 08:57:15.056]: Info: -- Output --
[2026-08-20 08:57:15.056]: Info:   Resolution: 2560x1440
[2026-08-20 08:57:15.056]: Info:   Offset: 1920x0
[2026-08-20 08:57:15.075]: Error: Couldn't release NvFBC context from current thread:
[2026-08-20 08:58:14.980]: Info: Executing [Desktop]
[2026-08-20 08:58:15.071]: Info: New streaming session started [active sessions: 1]
[2026-08-20 08:58:15.106]: Info: CLIENT CONNECTED
[2026-08-20 08:58:15.350]: Info: Found [2] outputs
[2026-08-20 08:58:15.350]: Info: Virtual Desktop: 4480x1440
[2026-08-20 08:58:15.350]: Info: XrandR: available
[2026-08-20 08:58:15.350]: Info: -- Output --
[2026-08-20 08:58:15.350]: Info:   Resolution: 1920x1080
[2026-08-20 08:58:15.350]: Info:   Offset: 0x0
[2026-08-20 08:58:15.350]: Info: -- Output --
[2026-08-20 08:58:15.350]: Info:   Resolution: 2560x1440
[2026-08-20 08:58:15.350]: Info:   Offset: 1920x0
[2026-08-20 08:58:15.368]: Error: Couldn't release NvFBC context from current thread:
[2026-08-20 08:58:15.369]: Info: Screencasting with NvFBC
[2026-08-20 08:58:15.464]: Info: Creating encoder [hevc_nvenc]

(Client then sees output 0's content, not output 1's — despite output_name=1 and this log confirming output 1's correct geometry was resolved both at startup and again at stream-session start.)

Start with src/platform/linux/cuda.cpp, especially the handle_t/display_t NVFBC path, and compare its capture-to-encoder flow with the standalone NvFBC.h reproduction. Reproduce the dual-head configuration using output_name=1 and trace the frame after NvFBCToCudaGrabFrame or NvFBCToSysGrabFrame. Done means the stream displays DP-0 content rather than HDMI-0 content.