KhronosGroup / KhronosGroup/Vulkan-ValidationLayers

GPU-AV causes GPU hang

Open
#12,525 9 comments 0 reactions 1 assignee Claimed by @spencer-lunarg View on GitHub
GPU-AV
Dominant language
C++
Stars
1k
Forks
504
Avg merge
11h 35m
Merged PRs (30d)
224

Description

**Environment:**
- OS: Windows 11 and Linux
- GPU and driver version: RTX 5080, Windows: 610.62, 596.36; Linux: 595.84
- SDK or header version if building from repo: 1.4.350.1, also tried validation layers build from commit `38d4f2d9cacbce5a0e46e756792e3a3cae7ebe28`
- Options enabled (synchronization, best practices, etc.): GPU-AV with shader instrumentation

**Describe the Issue**
Program which passes normal validation layers without any errors causes GPU hang when GPU-AV is enabled. I use VK_EXT_desciptor_heap but the problematic part seems to be vertex fetch via BDA in vertex shader.
The crash doesn't happen all the time, it seems like every memory access in problematic shader has a chance of crashing as increasing number of vertices drawn increases chance of crash.

Enabling safety features of GPU-AV doesn't help.

Shader code below has a few commented lines.
`MeshVertex vertex = {}` disables vertex fetching and it seems like it makes the crash disappear
`const MeshVertex vertex = params.vertices[0];` crashes
`const MeshVertex vertex = params.vertices[min(vertex_id, 1000000)]` crashes more frequently

I tested it with passing GPU address of empty buffer with 1000001 * 16 bytes so it shouldn't be caused by OOB access.
What is interesting is that I have very similar shader for ImGui which never crashes.

What could possibly go wrong in this shader?
- accessing index buffer: this seems unlikely as without derefing vertices it doesn't crash
- accessing vertex buffer: probably this but why? The buffer is allocated with static size so no OOB access. With full shader and no GPU-AV the geometry rendered looks perfectly fine
- accessing depth: problems should be caught by normal validation

**Expected behavior**
Program runs with or without validation errors but GPU doesn't hang.

**Additional context**

code

```cpp
// This probably doesn't matter as there are no descriptor heap accesses in this shader.
[vk::binding(0, 8)]
__DynamicResource<__DynamicResourceKind.General> __resource_descriptor_heap[];

[vk::binding(0, 9)]
__DynamicResource<__DynamicResourceKind.Sampler> __sampler_descriptor_heap[];

[ForceInline]
export T getDescriptorFromHandle(DescriptorHandle handle) where T : IOpaqueDescriptor {
if (T.kind == DescriptorKind.CombinedTextureSampler || T.kind == DescriptorKind.AccelerationStructure) {
return defaultGetDescriptorFromHandle(handle);
}

__target_switch {
case spirv: {
if (T.kind == DescriptorKind.Sampler) {
return __sampler_descriptor_heap[((uint2)handle).x].asOpaqueDescriptor();
} else {
return __resource_descriptor_heap[((uint2)handle).x].asOpaqueDescriptor();
}
}
default: {
return defaultGetDescriptorFromHandle(handle);
}
}
}

struct MeshVertex {
float px, py, pz, pp;
};
struct Scene {
};
struct ShaderParameters {
Scene* scene;
MeshVertex* vertices;
};
[[vk::push_constant]] uniform ShaderParameters params;

struct RasterizerData {
float4 position : SV_Position;
};

[shader("vertex")]
RasterizerData vs_main(uint vertex_id : SV_VulkanVertexID) {
const MeshVertex vertex = params.vertices[min(vertex_id, 1000000)];
// const MeshVertex vertex = params.vertices[0];
// MeshVertex vertex = {};
const var world_position = float4(float3(vertex.px, vertex.py, vertex.pz), 1.f);

RasterizerData output;
output.position = world_position;
return output;
}
```

SPIR-V:
```
; SPIR-V
; Version: 1.5
; Generator: Khronos Slang Compiler; 0
; Bound: 43
; Schema: 0
OpCapability PhysicalStorageBufferAddresses
OpCapability Shader
OpExtension "SPV_KHR_physical_storage_buffer"
%1 = OpExtInstImport "GLSL.std.450"
OpMemoryModel PhysicalStorageBuffer64 GLSL450
OpEntryPoint Vertex %main "main" %params %gl_Position %gl_VertexIndex
OpSource Slang 1
OpName %ShaderParameters_natural "ShaderParameters_natural"
OpMemberName %ShaderParameters_natural 0 "scene"
OpMemberName %ShaderParameters_natural 1 "vertices"
OpName %params "params"
OpName %MeshVertex_natural "MeshVertex_natural"
OpMemberName %MeshVertex_natural 0 "px"
OpMemberName %MeshVertex_natural 1 "py"
OpMemberName %MeshVertex_natural 2 "pz"
OpMemberName %MeshVertex_natural 3 "pp"
OpName %world_position "world_position"
OpName %main "main"
OpDecorate %_ptr_PhysicalStorageBuffer_uint ArrayStride 4
OpDecorate %_ptr_PhysicalStorageBuffer_MeshVertex_natural ArrayStride 16
OpDecorate %ShaderParameters_natural Block
OpMemberDecorate %ShaderParameters_natural 0 Offset 0
OpMemberDecorate %ShaderParameters_natural 1 Offset 8
OpDecorate %gl_VertexIndex BuiltIn VertexIndex
OpMemberDecorate %MeshVertex_natural 0 Offset 0
OpMemberDecorate %MeshVertex_natural 1 Offset 4
OpMemberDecorate %MeshVertex_natural 2 Offset 8
OpMemberDecorate %MeshVertex_natural 3 Offset 12
OpDecorate %gl_Position BuiltIn Position
%void = OpTypeVoid
%12 = OpTypeFunction %void
%float = OpTypeFloat 32
%v4float = OpTypeVector %float 4
%uint = OpTypeInt 32 0
%_ptr_PhysicalStorageBuffer_uint = OpTypePointer PhysicalStorageBuffer %uint
OpTypeForwardPointer %_ptr_PhysicalStorageBuffer_MeshVertex_natural PhysicalStorageBuffer
%ShaderParameters_natural = OpTypeStruct %_ptr_PhysicalStorageBuffer_uint %_ptr_PhysicalStorageBuffer_MeshVertex_natural
%_ptr_PushConstant_ShaderParameters_natural = OpTypePointer PushConstant %ShaderParameters_natural
%int = OpTypeInt 32 1
%int_1 = OpConstant %int 1
%_ptr_PushConstant_10 = OpTypePointer PushConstant %_ptr_PhysicalStorageBuffer_MeshVertex_natural
%_ptr_Input_int = OpTypePointer Input %int
%uint_1000000 = OpConstant %uint 1000000
%MeshVertex_natural = OpTypeStruct %float %float %float %float
%v3float = OpTypeVector %float 3
%float_1 = OpConstant %float 1
%_ptr_Output_v4float = OpTypePointer Output %v4float
%_ptr_PhysicalStorageBuffer_MeshVertex_natural = OpTypePointer PhysicalStorageBuffer %MeshVertex_natural
%params = OpVariable %_ptr_PushConstant_ShaderParameters_natural PushConstant
%gl_VertexIndex = OpVariable %_ptr_Input_int Input
%gl_Position = OpVariable %_ptr_Output_v4float Output
%uint_0 = OpConstant %uint 0
%_ptr_PhysicalStorageBuffer_float = OpTypePointer PhysicalStorageBuffer %float
%uint_1 = OpConstant %uint 1
%uint_2 = OpConstant %uint 2
%main = OpFunction %void None %12
%29 = OpLabel
%30 = OpAccessChain %_ptr_PushConstant_10 %params %int_1
%31 = OpLoad %int %gl_VertexIndex
%32 = OpBitcast %uint %31
%33 = OpExtInst %uint %1 UMin %32 %uint_1000000
%34 = OpLoad %_ptr_PhysicalStorageBuffer_MeshVertex_natural %30
%35 = OpPtrAccessChain %_ptr_PhysicalStorageBuffer_MeshVertex_natural %34 %33
%36 = OpAccessChain %_ptr_PhysicalStorageBuffer_float %35 %uint_0
%37 = OpLoad %float %36 Aligned 4
%38 = OpAccessChain %_ptr_PhysicalStorageBuffer_float %35 %uint_1
%39 = OpLoad %float %38 Aligned 4
%40 = OpAccessChain %_ptr_PhysicalStorageBuffer_float %35 %uint_2
%41 = OpLoad %float %40 Aligned 4
%42 = OpCompositeConstruct %v3float %37 %39 %41
%world_position = OpCompositeConstruct %v4float %42 %float_1
OpStore %gl_Position %world_position
OpReturn
OpFunctionEnd
```

Compilation command line:
```
slang
-lang slang
-std 2026
-target spirv
-profile spirv_1_5
-fvk-use-scalar-layout
-default-image-format-unknown
-capability spvRayQueryKHR
-O3
shader.slang
-stage vertex
-D vs_main=main
-o output.bin
```

Descriptor heap setup (probably not relevant here):
```cpp
for (size_t i = 0; i < std::size(m_resource_heap_views); ++i) {
uint32_t binding_count{};
VkSpirvResourceTypeFlagsEXT resource_mask{};
VkDescriptorMappingSourceConstantOffsetEXT constant_offset{};

if (i == 0) {
binding_count = resource_heap.entry_count();
resource_mask = VK_SPIRV_RESOURCE_TYPE_SAMPLED_IMAGE_BIT_EXT |
VK_SPIRV_RESOURCE_TYPE_READ_ONLY_IMAGE_BIT_EXT |
VK_SPIRV_RESOURCE_TYPE_READ_WRITE_IMAGE_BIT_EXT |
VK_SPIRV_RESOURCE_TYPE_UNIFORM_BUFFER_BIT_EXT |
VK_SPIRV_RESOURCE_TYPE_READ_ONLY_STORAGE_BUFFER_BIT_EXT |
VK_SPIRV_RESOURCE_TYPE_READ_WRITE_STORAGE_BUFFER_BIT_EXT;
constant_offset = {
.heapOffset = 0,
.heapArrayStride = uint32_t(resource_heap.entry_size()),
};
} else {
binding_count = sampler_heap.entry_count();
resource_mask = VK_SPIRV_RESOURCE_TYPE_SAMPLER_BIT_EXT;
constant_offset = {
.heapOffset = 0,
.heapArrayStride = uint32_t(sampler_heap.entry_size()),
};
}

// samplerHeapOffset is used only when mapping a combined image sampler, used in place of
// heapOffset to retrieve the sampler.
// We can ignore it.

m_resource_heap_views[i] = {
.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_SET_AND_BINDING_MAPPING_EXT,
.descriptorSet = 8 + uint32_t(i),
.firstBinding = 0,
.bindingCount = binding_count,
.resourceMask = resource_mask,
.source = VK_DESCRIPTOR_MAPPING_SOURCE_HEAP_WITH_CONSTANT_OFFSET_EXT,
.sourceData = {
.constantOffset = constant_offset,
},
};
}
m_descriptor_set_and_binding_mapping = {
.sType = VK_STRUCTURE_TYPE_SHADER_DESCRIPTOR_SET_AND_BINDING_MAPPING_INFO_EXT,
.mappingCount = uint32_t(std::size(m_resource_heap_views)),
.pMappings = m_resource_heap_views
};
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.