xpu-smi --query-gpu during Vulkan llama.cpp inference causes VK_ERROR_DEVICE_LOST on Arc B390 (Windows)

Hi!

While developing a feature to display GPU utilization in https://github.com/intel/AI-Playground/ we stumbled upon an issue where querying the information via xpu-smi during inference via llama.cpp (Vulkan / Windows) caused the device to crash. Below is an AI generated summary and reproducer. Let me know if you need more info or if this is too far outside the xpu-smi use-case.

Best,
Markus

## Summary

Polling GPU metrics with `xpu-smi --query-gpu=…` while llama.cpp is running a Vulkan decode on the same Intel Arc GPU causes a Vulkan device lost. llama.cpp then fails with:

```
E ggml_vulkan: device lost on Vulkan0
E ggml_vulkan: getFenceStatus at ggml/src/ggml-vulkan/ggml-vulkan.cpp:2646
E srv update_slots: decode() failed: vk::Device::getFenceStatus: ErrorDeviceLost
```

`xpu-smi` itself keeps running and prints plausible CSV. The inference process is the one that dies.

The same inference completes if `xpu-smi` is not running. Windows PDH GPU Engine counters (`\GPU Engine(*)\Utilization Percentage`) polled on the same cadence also do **not** trigger the loss — only the Level Zero `xpu-smi` query does.

No Windows System TDR (event 4101) is logged. The loss is visible to Vulkan only.

## Environment

| | |
| --- | --- |
| OS | Windows 11 Pro 10.0.26200 |
| GPU | Intel Arc B390 |
| Graphics driver | 32.0.101.8974 |
| xpu-smi | 2.1.0.20250225 (`8389eee7`) |
| Level Zero | 1.32.0 |
| llama.cpp | win-vulkan-x64, `0.3.0-dev` build 10666 (`4e97ac86e`) |
| Model | Qwen3.5-9B-Q4_K_M.gguf, `--gpu-layers 999 --ctx-size 32000 -fa on --no-mmap --cache-ram 16384` |

## How to reproduce

Two processes only: stock `llama-server` and a 2-second `xpu-smi` loop. No other app.

1. Start llama.cpp (Vulkan):

```bat
llama-server.exe --model Qwen3.5-9B-Q4_K_M.gguf --port 39110 --ctx-size 32000 --gpu-layers 999 --log-prefix --jinja --no-mmap -fa on --cache-ram 16384 --host 127.0.0.1
```

2. Wait until `GET http://127.0.0.1:39110/health` returns `{"status":"ok"}`.

3. In another terminal, poll every 2s (this is the query this xpu-smi 2.1 build accepts):

```bat
:loop
xpu-smi.exe --query-gpu=index,name,utilization.gpu,memory.used,memory.total,power.draw,clocks.current.graphics --format=csv,noheader,nounits
timeout /t 2 /nobreak >nul
goto loop
```

4. Send a generation long enough to overlap several polls (~12s / ~200 tokens is enough):

```bat
curl http://127.0.0.1:39110/v1/chat/completions -H "Content-Type: application/json" -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Write a complete single-file HTML and JavaScript tic-tac-toe game.\"}],\"max_tokens\":1024,\"temperature\":0.7}"
```

**Expected:** completion finishes (~19 t/s on this box).

**Actual:** around 160–190 generated tokens (~12s into decode) llama-server returns HTTP 500:

```json
{"error":{"code":500,"message":"decode() failed: vk::Device::getFenceStatus: ErrorDeviceLost","type":"server_error"}}
```

## Isolation

Same `llama-server`, same prompt, 2s poll interval:

| Concurrent sampler | Result |
| --- | --- |
| none (control) | OK, 512 tokens @ ~19.6 t/s |
| Windows PDH `\GPU Engine(*)\Utilization Percentage` only | OK, 1024 tokens |
| `xpu-smi --query-gpu=…` on the **first** request after a fresh server | **device-lost at ~160–190 tokens** |
| `xpu-smi --query-gpu=…` on a later request | **device-lost** |

While the query is running, `xpu-smi` reports the card busy (example: util ~14%, power ~24 W, clock 2500 MHz, memory.used ~35 GiB on this iGPU). After the Vulkan loss, later PDH engine rows drop to 0.

Start by reproducing the failure with the documented llama-server Vulkan command and the 2-second xpu-smi --query-gpu loop, then compare it with the Windows PDH control. Trace the xpu-smi query entry point and its Level Zero interaction while Vulkan inference runs; done means the completion finishes without VK_ERROR_DEVICE_LOST during polling.

Written by the indexing model from the issue text.