KhronosGroup / KhronosGroup/Vulkan-ValidationLayers

Vulkan video decode crashes with unique_handles validation on NVIDIA

Open
#12,151 2 comments 0 reactions 0 assignees View on GitHub
Video
Dominant language
C++
Stars
1k
Forks
504
Avg merge
11h 35m
Merged PRs (30d)
224

Description

## Summary

With Vulkan SDK 1.4.341.1 validation layers and NVIDIA driver 595.58.03 on Linux, FFmpeg Vulkan H.264 decode segfaults inside libnvidia-glcore.so during vkQueueSubmit2 when VK_LAYER_KHRONOS_validation is enabled.

The crash reproduces with a public FFmpeg sample:

https://samples.ffmpeg.org/V-codecs/h264/missing_frames.avi

The crash requires validation core plus unique handle wrapping:

```text
khronos_validation.validate_core = true
khronos_validation.unique_handles = true
```

The same workload succeeds when `unique_handles=false` while keeping standard validation checks otherwise enabled. No validation message is emitted before the segfault.

## Repro

Download sample:

```bash
curl -L -o missing_frames.avi https://samples.ffmpeg.org/V-codecs/h264/missing_frames.avi
```

Run FFmpeg Vulkan decode with validation enabled:

```bash
VK_LAYER_PATH=/home/aim/vulkan/1.4.341.1/x86_64/share/vulkan/explicit_layer.d \
VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation \
./ffmpeg -v warning -threads 1 \
-init_hw_device vulkan=vk \
-hwaccel vulkan -hwaccel_output_format vulkan \
-i missing_frames.avi \
-frames:v 8 -f null -
```

Result:

```text
Segmentation fault (core dumped)
exit status 139
```

## Stack excerpt

```text
Thread received signal SIGSEGV
#0 libnvidia-glcore.so.595.58.03
#9 vvl::dispatch::Device::QueueSubmit2
#10 vulkan_layer_chassis::QueueSubmit2
#11 ff_vk_exec_submit
#12 ff_vk_decode_frame
```

## Isolation

Passing case: standard validation settings with only unique_handles disabled:

```text
khronos_validation.validate_core = true
khronos_validation.unique_handles = false
```

Failing minimal combination:

```text
khronos_validation.validate_core = true
khronos_validation.unique_handles = true
```

Other checks tested individually did not reproduce the crash.

## Environment

- Vulkan SDK / validation layers: 1.4.341.1
- NVIDIA driver: 595.58.03
- GPU: NVIDIA GeForce RTX 5080
- OS: Linux (Omarchy)
- Application: FFmpeg master, Vulkan H.264 hardware decode
- Public sample: https://samples.ffmpeg.org/V-codecs/h264/missing_frames.avi

## Notes

This looks like a unique-handle wrapping/unwrapping issue around Vulkan Video objects or submit-time validation. The command stream runs without validation, and it also runs with validation when unique_handles is disabled.

Contributor guide

Open the contributing guide

Research direction

Reproduce the crash with missing_frames.avi using the provided FFmpeg Vulkan command and validation settings. Compare the failing unique_handles=true case with unique_handles=false, then trace the submit path through vvl::dispatch::Device::QueueSubmit2, vulkan_layer_chassis::QueueSubmit2, ff_vk_exec_submit, and ff_vk_decode_frame. Done means the workload no longer segfaults with unique-handle wrapping enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
computer-graphics
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.