CUDA: illegal memory access in indexer_topk_stream512_kernel on prefills above ~32k tokens (sm_120)

## Summary

On a single-GPU CUDA build, writing a `continued` KV disk checkpoint **while a prefill
is still running** corrupts the CUDA context. The failing request aborts, and every
later request in the same process fails too — `ds4-server` keeps answering
`/v1/models` with 200 but can no longer run inference. Only a restart recovers it.

Workaround that has held so far: `--kv-cache-continued-interval-tokens 200000`,
i.e. never checkpoint mid-prefill.

## Environment

| | |
|---|---|
| ds4 | `84cc882352757baf628a1776badf7cc54d584e28` (2026-08-09, "rocm: enable DSpark speculative decoding") |
| Build | `make cuda CUDA_ARCH=sm_120` |
| GPU | NVIDIA RTX PRO 6000 Blackwell Workstation Edition, 96 GB, driver 610.57.04 |
| CUDA | 13.3, V13.3.73 |
| OS | Linux 7.1.8-1-cachyos (Arch) |
| Model | `ds4f-q2` (`DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2-imatrix-0731.gguf`) |

Server started as:

```
DS4_CUDA_Q8_F16_CACHE_MB=2048 DS4_CUDA_Q8_F16_CACHE_RESERVE_MB=3072 \
./ds4-server --cuda --gpu-vram 86 --prefill-chunk 1024 --ctx 131072 \
--host 0.0.0.0 --port 8081 \
--kv-disk-dir ~/.ds4/server-kv --kv-disk-space-mb 204800
```

## What happens

A cold 77,832-token prefill runs at ~1,700 t/s. At 30,720 tokens the periodic
`continued` checkpoint is written (default `--kv-cache-continued-interval-tokens
10000`). The chunk that overlaps the save drops to 915 t/s, and seven seconds
later the context dies:

```
12:58:53 prefill chunk 30720/77832 (39.5%) chunk=1717.21 t/s avg=1688.61 t/s
12:58:54 kv cache stored tokens=30720 trimmed=0 reason=continued key=token-text size=426.26 MiB save=202.7 ms
12:58:55 prefill chunk 31744/77832 (40.8%) chunk=915.77 t/s avg=1643.86 t/s <-- overlaps the save
12:58:56 prefill chunk 33792/77832 (43.4%) chunk=1732.12 t/s avg=1648.15 t/s
12:58:58 prefill chunk 37888/77832 (48.7%) chunk=1680.85 t/s avg=1651.78 t/s
ds4: CUDA end commands failed: an illegal memory access was encountered
```

After that first error, **every** subsequent request fails, including tiny ones:

```
12:59:29 chat ctx=0..2104:2104 prompt start
ds4: CUDA tensor fill f32 launch failed: an illegal memory access was encountered
12:59:33 chat ctx=0..314:314 prompt start
ds4: CUDA tensor fill f32 launch failed: an illegal memory access was encountered
```

43 such lines before I restarted. The HTTP layer stays up, so a client only sees
`Server error: cuda prefill failed` and cannot tell that the process is unusable.

## A second, probably related failure

Earlier the same day, with the same build and `--kv-disk-space-mb 65536`, an
attempt to extend an existing checkpoint failed while the cache was evicting
entries:

```
12:38:47 kv cache evicted reason=disk-cache-full tokens=20480 hits=0 size=291.80 MiB
12:38:48 kv cache stored tokens=30720 trimmed=0 reason=continued size=426.26 MiB
ds4: CUDA synchronize failed: an illegal memory access was encountered
12:38:53 kv cache discarded reason=prefill-failed file=.../fef29a254eda2c6b8157eb5002bc695c3b6757dc.kv
```

The client saw `cuda resumed prefill failed while extending checkpoint`. Here the
server recovered by discarding the checkpoint; in the first case it did not.

## Two smaller observations

**Metal messages on a CUDA build.** The error path prints Metal diagnostics on a
CUDA-only build, which is confusing when reading logs:

```
ds4: Metal synchronize after whole-prefill graph failure also failed
ds4: Metal synchronize after chunked prefill failure also failed
```

**A 96 GB card needs undocumented env vars to load `ds4f-q2`.** The README lists
Flash Q2 as fitting 96 GB machines, but with `--gpu-vram auto` the session
creation OOMs: the Q8→F16 dequant cache grows into whatever is free (it had taken
5.21 GiB when allocation failed at `free=0.04 GiB`). It only works after capping
it with `DS4_CUDA_Q8_F16_CACHE_MB` and raising
`DS4_CUDA_Q8_F16_CACHE_RESERVE_MB`, neither of which appears in `--help`. A line
in the README, or an automatic cap when the budget is close to total VRAM, would
save others the search through `ds4_cuda.cu`.

## What I can offer

Full logs of both crashes are kept and can be attached. I can also test patches
on this hardware — the machine is a single-GPU sm_120 workstation that reproduces
the first crash reliably within a few large prefills when
`--kv-cache-continued-interval-tokens` is left at its default.