ALVR: Errors with 'Failed to create VAAPI encoder'

Nixpkgs version

• Unstable (26.05)

Describe the bug

ALVR starts fine, but trying to use it results in the following error messages:

Failed to create VAAPI encoder: filter_in creation failed: Invalid argument
Please make sure you have installed VAAPI runtime

Encoder: Setting BufferSourceContext.pix_fmt to a HW format requires hw_frames_ctx to be non-NULL!

Steps to reproduce

1. Run ALVR
2. Try to connect from a HMD (Quest 3 in my case)
3. Witness visual artifacts as the connection dies and receive errors on the desktop

Expected behaviour

It doesn't error

Screenshots

No response

Relevant log output

Additional context

Was first reported upstream https://github.com/alvr-org/ALVR/issues/3134

ALVR version 20.14.1 on 25.11/unstable appears to have this behavior. Reproduced with various kernels including default. 20.13.0 on 25.05 seemed to work fine for me.

System metadata

 - system: `"x86_64-linux"`
 - host os: `Linux 6.18.0-cachyos, NixOS, 26.05 (Yarara), 26.05.20251208.addf7cf`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.31.2+1`
 - channels(root): `"nixos-24.05"`
 - nixpkgs: `/nix/store/98mad1s6hrryyflb6qf6hd4xyb8r7cy4-source`

Note: I also tried 25.11 stable as well as the latest 6.18.0 kernel from nixpkgs

Notify maintainers

@LuNeder
@jopejoe1

---

Note for maintainers: Please tag this issue in your pull request description. (i.e. Resolves #ISSUE.)

I assert that this issue is relevant for Nixpkgs

• I assert that this is a bug and not a support request.
• I assert that this is not a duplicate of an existing issue.
• I assert that I have read the NixOS Code of Conduct and agree to abide by it.

Is this issue important to you?

Add a 👍 reaction to issues you find important.

Start with upstream ALVR issue #3134, then inspect the nixpkgs ALVR package definition and compare the 20.13.0 and 20.14.1 packaging or runtime inputs. Reproduce the Quest connection failure on the affected NixOS versions and verify that ALVR connects without VAAPI encoder errors or visual artifacts.