Steam remote play hardware encoding doesn't works: NVENC - Couldn't create encoding session: 15

**Distribution name and version where applicable**

```bash
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

$ uname -a
Linux debian 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

$ vainfo
libva info: VA-API version 1.17.0
libva error: vaGetDriverNameByIndex() failed with unknown libva error, driver_name = (null)
vaInitialize failed with error code -1 (unknown libva error),exit

$ vainfo --display drm
libva info: VA-API version 1.17.0
libva info: Trying to open /usr/lib/x86_64-linux-gnu/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_17
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.17 (libva 2.12.0)
vainfo: Driver version: Mesa Gallium driver 22.3.6 for AMD Radeon Graphics (gfx1036, LLVM 15.0.6, DRM 3.49, 6.1.0-18-amd64)
vainfo: Supported profile and entrypoints
VAProfileH264ConstrainedBaseline: VAEntrypointVLD
VAProfileH264ConstrainedBaseline: VAEntrypointEncSlice
VAProfileH264Main : VAEntrypointVLD
VAProfileH264Main : VAEntrypointEncSlice
VAProfileH264High : VAEntrypointVLD
VAProfileH264High : VAEntrypointEncSlice
VAProfileHEVCMain : VAEntrypointVLD
VAProfileHEVCMain : VAEntrypointEncSlice
VAProfileHEVCMain10 : VAEntrypointVLD
VAProfileHEVCMain10 : VAEntrypointEncSlice
VAProfileJPEGBaseline : VAEntrypointVLD
VAProfileVP9Profile0 : VAEntrypointVLD
VAProfileVP9Profile2 : VAEntrypointVLD
VAProfileAV1Profile0 : VAEntrypointVLD
VAProfileNone : VAEntrypointVideoProc

$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05 Driver Version: 525.147.05 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
```

**Steam Client version**

Both the normal version and the `Steam Beta Update` have this issue. Currently using `Steam Beta Update`.

**Flatpak info**

```bash
$ flatpak --version
Flatpak 1.14.4

$ flatpak --gl-drivers
nvidia-525-147-05
default
host
```

**GPU info**

```bash
$ lspci | grep VGA
01:00.0 VGA compatible controller: NVIDIA Corporation AD103 [GeForce RTX 4080] (rev a1)
16:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raphael (rev c1)
```

The second one is iGPU of AMD 7950x; Display connect to NVIDIA GPU.

**Problem description**

1. Open steam use command: `flatpak run com.valvesoftware.Steam`
2. Enable all hardware encoding in remote play setting page.
3. Open `Steam Link` on any device or use `Steam Remote Play Together`
4. Start streaming and monitor steam logs in the terminal.

Then I got this:

```log
Streaming initialized and listening on port 27031
tid(551) burning pthread_key_t == 0 so we never use it
02/15 17:34:37 Init: Installing breakpad exception handler for appid(steam_monitor)/version(20240214222356)/tid(551)
02/15 17:34:37 Init: Installing breakpad exception handler for appid(steam_monitor)/version(1.0)/tid(551)
CLIENT: Sending HID device 0000/11fb/-1 Mobile Touch Control at touch://0
Streaming started to XXXXXXXXX Android 14 at 192.168.191.7:43121, audio channels = 2, MTU = 1468
Streaming quality: k_EStreamQualityBalanced
Streaming bitrate: Automatic
Device form factor: phone
Maximum capture: 2400x1080 60.00 FPS
Video Streaming: enabled
Audio Streaming: enabled
Input Streaming: enabled

=====================================================================
Game: UNKNOWN (0)
Setting target bitrate to 15000 Kbit/s, burst bitrate is 75000 Kbit/s
GameScope focus changed to appID 0
Window title set to ~ : bwrap — Yakuake
Changing record window: 0x3c0000a
>>> Starting desktop stream
SynchronizeClientState(): setting capture size 2560x1440
SynchronizeClientState(): setting title to ~ : bwrap — Yakuake
SynchronizeClientState(): setting icon 24x24
SynchronizeClientState(): setting activity to k_EStreamActivityDesktop: Steam Controller Configs - Desktop
Recording system audio
PulseAudio: Sink alsa_output.pci-0000_01_00.1.hdmi-stereo is default
PulseAudio: Adding sink[0] alsa_output.pci-0000_01_00.1.hdmi-stereo with priority 3, monitor is alsa_output.pci-0000_01_00.1.hdmi-stereo.monitor
PulseAudio: Recording on device alsa_output.pci-0000_01_00.1.hdmi-stereo.monitor
Audio mix: start=447398451, returned=0
Audio source [System Pulse]: init=0, adjustment=0, through=0, last_start=0, mixed=0, drop_before=0, drop_after=0
PulseAudio: Connected to device alsa_output.pci-0000_01_00.1.hdmi-stereo.monitor (0, not suspended).
PulseAudio: Using sample spec 's16le 2ch 48000Hz', channel map 'front-left,front-right'.
>>> Switching video stream from NONE to Desktop_MovieStream
BuildCompleteAppOverviewChange: 500 apps
NVENC - Couldn't create encoding session: 15
NVENC - Couldn't create encoding session: 15
CGameStreamVideoStageVAAPI: Reinitializing 1920x1080 at 15000 Kbit/s and 59.75 FPS with quality 0
ffmpeg verbose: libva: VA-API version 1.19.0
ffmpeg verbose: libva: Trying to open /usr/lib/i386-linux-gnu/dri/nvidia_drv_video.so
ffmpeg verbose: libva: Trying to open /usr/lib/i386-linux-gnu/dri/intel-vaapi-driver/nvidia_drv_video.so
ffmpeg verbose: libva: Trying to open /usr/lib/i386-linux-gnu/GL/lib/dri/nvidia_drv_video.so
ffmpeg verbose: libva: va_openDriver() returns -1
ffmpeg error: Failed to initialise VAAPI connection: -1 (unknown libva error).
CGameStreamVideoStageVAAPI: Failed to create device context: Input/output error

Detected 32 logical processors, using 4 threads
Configuring encoder: [threads=4][width=1920][height=1080][preset=superfast][tune=zerolatency]
Video Encoder Changed
>>> Capture method set to Desktop OpenGL NV12 + libx264 main (4 threads)
>>> Capture resolution set to 1920x1080
>>> Capture colorspace set to BT.601 limited range
```

I noticed this line:

```log
NVENC - Couldn't create encoding session: 15
```

then steam check VAAPI, but x11 is running on NVIDIA GPU:

```log
ffmpeg verbose: libva: Trying to open /usr/lib/i386-linux-gnu/dri/nvidia_drv_video.so
ffmpeg verbose: libva: Trying to open /usr/lib/i386-linux-gnu/dri/intel-vaapi-driver/nvidia_drv_video.so
ffmpeg verbose: libva: Trying to open /usr/lib/i386-linux-gnu/GL/lib/dri/nvidia_drv_video.so
ffmpeg verbose: libva: va_openDriver() returns -1
```

but x11 code path of VAAPI doesn't work, although there is an AMD iGPU.

Maybe steam can handle this situation in the future: [Steam Beta Adds VA-API DRM Video Acceleration For Remote Play](https://www.phoronix.com/news/Steam-Remote-Play-VA-API-DRM), but for now, I want to find out why nvenc doesn't work (I don't know how to force steam to use iGPU encoding).

I found a related issue here:

https://github.com/ValveSoftware/steam-for-linux/issues/10174

and the maintainer replied:

> It looks like you need to have the libnvidia-encode1 and libnvidia-encode1:i386 system packages installed to support NVENC on Debian. If adding those packages doesn't help, then report the broken hardware encode path to your distro's package maintainer(s) for the NVIDIA video driver.

**Does this issue reproduce with native Steam**

Have not tested