intel / intel/llvm

[SYCL] SYCL_CACHE_PERSISTENT=1 segfaults during first-compile JIT on Level-Zero V2 adapter (Xe2 + Xe3, 6/6 deterministic, clean cache)

Open
#22,853 2 comments 0 reactions 1 assignee Claimed by @uditagarwal97 View on GitHub
bug confirmed
Dominant language
LLVM
Stars
1.5k
Forks
854
Avg merge
3d 17h
Merged PRs (30d)
137

Description

### Describe the bug

Setting `SYCL_CACHE_PERSISTENT=1` causes a deterministic SIGSEGV during initial kernel JIT compilation in llama.cpp's SYCL backend (llama-server / llama-cli). The crash occurs during the persistent cache **write** path on first compile — including from a completely clean cache state on a fresh filesystem, so it is not the known stale-cache/changed-binary failure mode. Reproduced 6/6 across:

- three application builds (Ollama v0.32.5's bundled llama-server via its scheduler; the same binary invoked directly; the official `ghcr.io/ggml-org/llama.cpp:full-intel` image),
- two toolchain builds (oneAPI 2026.1.0 / icpx 2026.1.0.20260617; whatever full-intel ships),
- two runtime userspace versions (sycl-ls reports `1.15.39122+11` and `1.15.38308+1`),
- two GPU architectures (Battlemage/Xe2 discrete, Panther Lake/Xe3 integrated).

The common factor across every crashing configuration is the **Level-Zero V2 unified runtime with the persistent-cache write path engaged**. Identical commands with the variable unset work normally on every configuration (JIT completes in ~15s).

Prior art in the reproducer application's tracker: ggml-org/llama.cpp#22095 (closed not-planned) and its predecessors #13674, #13536, #20368, #21474 — all closed without root cause. A ggml maintainer there reported hitting it once and being unable to reproduce; the reproduction below is on-demand.

### To reproduce

Fastest form — official llama.cpp Intel image, any gguf model, ~90 seconds:

```
docker run --rm --device /dev/dri \
-e ONEAPI_DEVICE_SELECTOR=level_zero:0 \
-e SYCL_CACHE_PERSISTENT=1 \
-v /path/to/models:/models:ro \
ghcr.io/ggml-org/llama.cpp:full-intel \
--run -m /models/.gguf -ngl 99 -n 10 -p "hello" -no-cnv
```

Dies during model load; exit code **139**. Remove the `SYCL_CACHE_PERSISTENT` line, rerun: loads and generates normally. Tested model: qwen2.5-3b Q4; not model-specific.

Full 6/6 matrix:

| # | Build | Device | Cache state at launch | Result |
|---|---|---|---|---|
| 1 | Ollama v0.32.5 bundled llama-server (oneAPI 2026.1 container, via Ollama scheduler) | Arc B390 iGPU (Panther Lake, Xe3) | fresh — dir did not exist | SIGSEGV ~2s |
| 2 | same | B390 (Xe3) | after container restart (partial cache present) | SIGSEGV ~2s |
| 3 | same | B390 (Xe3) | **guaranteed clean** — container removed + recreated, fresh filesystem | SIGSEGV ~2s |
| 4 | same | Arc Pro B70 (Battlemage G31, Xe2, eGPU) | guaranteed clean | SIGSEGV ~2s |
| 5 | same llama-server binary invoked directly (no orchestration) | B70 (Xe2) | pre-existing | SIGSEGV at **0.5s during model load** |
| 6 | official `full-intel` image (independent build/toolchain) | B70 (Xe2) | fresh in `--rm` container | SIGSEGV during load, exit 139 |

Cleanest stderr (row 5, direct invocation):

```
0.00.200.995 I srv load_model: loading model '...'
0.00.544.077 W load: control-looking token: 128247 '' was not control-type; ...
Segmentation fault (core dumped)
```

`$HOME/.cache/libsycl_cache/` is created and contains one hash-named subdirectory at the moment of death — the cache mechanism engages in the documented location, then dies during the write.

### Environment

- OS: Ubuntu 26.04 Desktop, kernel 7.0.0-28-generic, `xe` driver
- Host: GMKtec EVO-T2S — Intel Core Ultra X7 358H (Panther Lake), 64GB LPDDR5X
- Target devices (one pinned per run via `ONEAPI_DEVICE_SELECTOR=level_zero:`):
- Intel Arc B390 iGPU (Panther Lake, Xe3) — `sycl-ls`: `Intel(R) oneAPI Unified Runtime over Level-Zero V2 ... 30.0.4 [1.15.39122+11]`
- Intel Arc Pro B70 32GB (Battlemage G31, Xe2), eGPU via USB4 — same runtime line, `20.2.0 [1.15.39122+11]`
- DPC++ version (rows 1–5 container): oneAPI 2026.1.0, icpx 2026.1.0.20260617
- Userspace stack (rows 1–5): NEO 26.27.39122.11 · IGC 2.38.2+22051 · gmmlib 22.10.0 · libze1 1.32.0
- Row 6 (official image): independent build; its runtime reports `1.15.38308+1` (older NEO userspace) — crash reproduces there too
- Application: llama.cpp SYCL backend (via Ollama v0.32.5 and via the official full-intel image)

### Additional context

Diagnosis hypothesis: rows 3, 4, and 6 reproduce from a **virgin cache on a fresh filesystem**, ruling out old/new binary mix for this instance. What all runs share is many kernels being JIT-compiled **concurrently** at model load/warm-up — multiple threads hitting the persistent-cache write path simultaneously on first compile. Timing is consistent: the crash lands 0.5–2s into a JIT that takes ~15s with the flag off, i.e. during the first compilations, not after them.

I can provide full logs, core dumps, run instrumented builds, or test patches — the setup (Xe2 + Xe3 in one machine) reproduces this on demand.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.