KhronosGroup / KhronosGroup/Vulkan-ValidationLayers
GPU-AV: crash in Instruction::Describe() when an error record instruction offset is not an instruction boundary
- Dominant language
- C++
- Stars
- 1k
- Forks
- 504
- Avg merge
- 11h 35m
- Merged PRs (30d)
- 224
Description
**Environment:**
- OS: Arch Linux, kernel 7.2.3
- GPU and driver version: AMD Radeon RX 9070 XT (Navi 48, gfx1201), RADV, Mesa 26.2.2
- SDK or header version if building from repo: built from main at 1710dfc (the lines cited below are
unchanged on main as of c7178939a)
- Options enabled: GPU-AV (`VK_LAYER_GPUAV_ENABLE=1`)
**Describe the Issue**
GPU-AV can crash while formatting an error message, after validation has already done its job. The
offending instruction offset that GPU-AV writes into the error record is read back on the host and used
to decode a SPIR-V instruction, but nothing checks that the offset actually names one.
Building the layer with `-D_GLIBCXX_ASSERTIONS` and running llama.cpp's `test-backend-ops` under GPU-AV
aborts in `Instruction::Describe()`:
```
/usr/include/c++/16/bits/stl_vector.h:1372: ... back() [with _Tp = OperandKind]: Assertion '!this->empty()' failed.
#5 spirv::Instruction::Describe() at layers/state_tracker/shader_instruction.cpp:125
info = {types = std::vector of length 0, capacity 0}
remaining_words = 4294967295
#6 spirv::FindShaderSource (instructions = std::vector of length 18478,
instruction_position_offset = 7406)
at layers/error_message/spirv_logging.cpp:482
#7 SharedMemoryDataRace-RaceOnLoadStoreVsAtomic error logger (collide_inst_offset = 7406)
#8 gpuav::LogInstrumentationError
#9 gpuav::CommandBufferSubState::OnCompletion, QueueSubState::Retire, vvl::Queue::ThreadFunc
```
The mechanism, reading the frame locals:
1. Offset 7406 is inside the 18478 word module, so the existing
`collide_inst_offset < instrumented_shader->original_spirv.size()` check at
`shared_memory_data_race.cpp:109` passes. The offset is in range but does not sit on an instruction
boundary.
2. `Describe()` computes `remaining_words = length - operand_offset`, which underflows to 0xFFFFFFFF
because the word at that offset has a length field too small to be an instruction header.
3. The opcode decoded from that word is not in the generated grammar table, so `GetOperandInfo()` returns
the `OpNop` fallback at `spirv_grammar_helper.cpp:2874`, whose `types` vector is empty, and
`info.types.back()` on line 125 is then undefined behaviour.
A release build without libstdc++ assertions does not necessarily fault, which is probably why this has
gone unnoticed. Forcing a deliberately bad offset into the same path on a release build gives a plain
SIGSEGV in `Describe()` on the queue retire thread.
`Describe()` is only reached when the module has no OpLine or NonSemantic DebugInfo, so
`GetDebugLineOffset()` returns 0 and `FindShaderSource()` takes the "Unable to find shader source" branch
at `spirv_logging.cpp:478`. Shaders compiled without debug info, which is how llama.cpp ships its Vulkan
shaders, are the ones that reach it.
**How often the offset is bad**
I patched the data race error logger to walk the module and report whether each offset landed on an
instruction boundary, then ran one full `test-backend-ops` sweep. Excluding the 0 and 131071 values that
the pass uses as "not recorded" markers, 2001 shadow word offsets reached the logger:
| result | count |
|---|---|
| valid instruction boundary | 1986 |
| outside the module, caught by the existing size check | 14 |
| inside the module but mid instruction | 1 |
So roughly 1 in 2000, which is why it takes a long run to hit and why it looked state dependent at first.
Some of the bad shadow words look like float payloads rather than packed offsets, for example 0x3f800000
(1.0f) and 0xc44b8000 (-814.0f). I could not establish why, and it may be a separate question about the
pass rather than about the logging, but it suggests the slot was clobbered rather than merely stale.
**Expected behavior**
An offset written by an instrumented shader is data the host cannot trust. If it does not name a real
instruction, the message should degrade, for example by saying the instruction was not recorded, rather
than taking the process down inside the validation layer.
**Repro steps**
1. Build the layer from main with `-D_GLIBCXX_ASSERTIONS -g`.
2. Build llama.cpp with the Vulkan backend and run `test-backend-ops test` with
`VK_LAYER_GPUAV_ENABLE=1` and `GGML_VK_DISABLE_COOPMAT=1`.
3. It aborts somewhere in the FLASH_ATTN_EXT range once a SharedMemoryDataRace error is reported with a
bad shadow offset.
This is not a one command repro. The bad offset appears about once in 2000 reported offsets, so a full
sweep is needed, and it will not reproduce on a build without libstdc++ assertions even when the bad
offset occurs.
**Possible fix**
I have a candidate, 41 lines, if it is useful:
- an `IsInstructionBoundary()` helper in `spirv_logging.cpp` that walks the module and accepts the offset
only if it lands exactly on an instruction start whose length fits in what remains, used by all three
call sites that take an offset from an error record (`gpuav_shader_instrumentor.cpp:2197`,
`buffer_device_address.cpp:68`, `shared_memory_data_race.cpp:109`, the last of which currently has a
size check only). It runs only while building an error message.
- guards in `Describe()` for the empty operand list and for the `length < operand_offset` underflow.
With it, the same assertions build runs 18759 of 18759 backend tests clean while still reporting 15 data
race errors, and prints `(specific instruction not recorded)` for the 2 bad offsets it rejected in that
run. The full VVL test suite shows no change against baseline on this machine.
What I do not have is a positive test. The trigger is a shadow word written by the GPU, and I could not
find a way to forge one from the test framework. If you have a preferred way to test this kind of path, I
am happy to write it that way.
Contributor guide
Research direction
Start with spirv_logging.cpp, especially FindShaderSource(), then inspect spirv::Instruction::Describe() and the three error-record call sites named in the issue. Build with -D_GLIBCXX_ASSERTIONS and run llama.cpp's test-backend-ops plus the full VVL test suite. Done means malformed offsets no longer crash error formatting and the existing tests remain clean.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- devtools, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 54/100