VAAPI crashes at recording on Ubuntu

Operating System Info

Ubuntu 24.04

Other OS

No response

OBS Studio Version

32.1.1

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/DHs0eyObzTUgI5Dx

OBS Studio Crash Log URL

No response

Expected Behavior

GPU recording works (as in previous versions)

Current Behavior

GPU recording crashes the program

Steps to Reproduce

1. Select FFMPEG VAAPI encoder
2. Choose default settings
3. Do a simple scene
4. Start recording
   ...

Anything else we should know?

Here is a crash log with a backtrace of a debugging session on the flatpak version. This error happens in both the PPA and flatpak.

Thread 48 "obs gpu encode " received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff87fff6c0 (LWP 63)]

ff_avc_find_startcode_internal (p=0x0, end=0xfffffffffffffffd <error: Cannot access memory at address 0xfffffffffffffffd>) at /run/build/obs/libobs/obs-nal.c:28
28			if (p[0] == 0 && p[1] == 0 && p[2] == 1)
(gdb)
(gdb) bt
#0  ff_avc_find_startcode_internal
    (p=0x0, end=0xfffffffffffffffd <error: Cannot access memory at address 0xfffffffffffffffd>) at /run/build/obs/libobs/obs-nal.c:28
#1  obs_nal_find_startcode (p=0x0, end=end@entry=0x0)
    at /run/build/obs/libobs/obs-nal.c:62
#2  0x00007ffff59182ad in obs_avc_keyframe
    (data=<optimized out>, size=<optimized out>)
    at /run/build/obs/libobs/obs-avc.c:30
#3  0x00007fffadbcd61d in vaapi_encode_internal
    (enc=enc@entry=0x5555589b7a60, frame=<optimized out>, packet=packet@entry=0x7fff87ffe6a0, received_packet=received_packet@entry=0x7fff87ffe667)
    at /run/build/obs/plugins/obs-ffmpeg/obs-ffmpeg-vaapi.c:696
#4  0x00007fffadbcde09 in vaapi_encode_tex
    (data=0x5555589b7a60, texture=0x7fff87ffe700, pts=1, lock_key=<optimized out>, next_key=<optimized out>, packet=0x7fff87ffe6a0, received_packet=0x7fff87ffe667)
    at /run/build/obs/plugins/obs-ffmpeg/obs-ffmpeg-vaapi.c:805
#5  0x00007ffff596f37f in gpu_encode_thread (data=<optimized out>)
    at /run/build/obs/libobs/obs-video-gpu-encode.c:158
#6  0x00007ffff2a9b56a in start_thread (arg=<optimized out>)
    at pthread_create.c:448
#7  0x00007ffff2b1ee94 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

Reproduce the crash with the FFMPEG VAAPI encoder using the default settings, then start at vaapi_encode_internal in plugins/obs-ffmpeg/obs-ffmpeg-vaapi.c and follow the call into obs_avc_keyframe in libobs/obs-avc.c. Use the supplied backtrace and OBS log to investigate the null data passed to obs_nal_find_startcode. Done means recording starts without a crash on Ubuntu 24.04.