Low-VRAM CUDA SSD-streaming results: 8 GB expert budget outperforms 10 GB on RTX 5070 Ti Laptop

Related: #732

I tested DeepSeek V4 Flash with CUDA SSD streaming on a low-VRAM Blackwell laptop setup and wanted to share the results separately because the cache-budget behavior may be useful for tuning.

## Environment

- WSL2 / Ubuntu 24.04
- CUDA 13.3
- NVIDIA GeForce RTX 5070 Ti Laptop GPU
- 12 GB VRAM
- Intel Core Ultra 7 255HX
- 32 GB DDR5-5600 system RAM (2x16 GB)
- NVMe SSD
- ds4 built from current `main`
- CUDA target: `sm_120`

Build:

```bash
make cuda CUDA_ARCH=sm_120
```

## Initial smoke test

A minimal SSD-streaming test completed successfully:

```bash
./ds4 \
-m ds4flash.gguf \
--ssd-streaming \
--ssd-streaming-cache-experts 12GB \
--ctx 4096 \
--tokens 1 \
--nothink \
-p "Reply with exactly: OK"
```

The requested 12 GiB expert budget was automatically capped to 10 GiB and the model generated `OK`.

The runtime reported:

```text
cuda SSD streaming total expert budget 10.00 GiB
3.38 GiB prefill headroom
6.62 GiB dynamic cache
resident model: 0.99 GiB
planned total: 11.41 GiB
```

## MMQ prefill OOM

With a normal prompt and 100 output tokens, the default MMQ prefill path failed with:

```text
CUDA model arena alloc failed for moe down mmq (1792.00 MiB chunk): out of memory
CUDA end commands failed: an illegal memory access was encountered
CUDA synchronize failed: an illegal memory access was encountered
prompt processing failed: cuda prefill failed
```

This matches the behavior reported in #732.

Using:

```bash
DS4_CUDA_MMQ=0
```

made SSD streaming stable again.

## 10 GB expert budget

Command:

```bash
DS4_CUDA_MMQ=0 ./ds4 \
-m ds4flash.gguf \
--ssd-streaming \
--ssd-streaming-cache-experts 10GB \
--ctx 4096 \
--tokens 100 \
--nothink \
-p "Explain in Turkish, in a few sentences, what a large language model is."
```

Result:

```text
prefill: 1.74 t/s
generation: 0.65 t/s
```

Memory plan:

```text
KV: 0.39 GiB
buffers: 0.03 GiB
resident model: 0.99 GiB
expert cache: 6.62 GiB
prefill expert reserve: 3.38 GiB
planned total: 11.41 GiB
```

## 8 GB expert budget

I then reduced the expert budget:

```bash
DS4_CUDA_MMQ=0 ./ds4 \
-m ds4flash.gguf \
--ssd-streaming \
--ssd-streaming-cache-experts 8GB \
--ctx 4096 \
--tokens 100 \
--nothink \
-p "Explain in Turkish, in a few sentences, what a large language model is."
```

Result:

```text
prefill: 1.95 t/s
generation: 0.73 t/s
```

Memory plan:

```text
KV: 0.39 GiB
buffers: 0.03 GiB
resident model: 0.99 GiB
expert cache: 4.62 GiB
prefill expert reserve: 3.38 GiB
planned total: 9.40 GiB
```

So on this setup, the smaller 8 GB expert budget was actually faster than 10 GB:

| Expert budget | Prefill | Generation | Planned GPU memory |
|---|---:|---:|---:|
| 10 GB | 1.74 t/s | 0.65 t/s | 11.41 GiB |
| 8 GB | 1.95 t/s | 0.73 t/s | 9.40 GiB |

The 8 GB profile also left significantly more VRAM headroom.

A non-fatal warning remained in both cases:

```text
CUDA host registration skipped: out of memory
```

but inference completed successfully.

## OpenAI-compatible server

I also verified the same configuration with `ds4-server`:

```bash
DS4_CUDA_MMQ=0 ./ds4-server \
-m ds4flash.gguf \
--ssd-streaming \
--ssd-streaming-cache-experts 8GB \
--ctx 4096 \
--host 0.0.0.0
```

The `/v1/models` endpoint worked correctly and the server was successfully connected to an Open WebUI instance.

The model responded successfully through the UI, including reasoning output.

On this hardware, however, end-to-end latency is still high enough that the setup is more functional / experimental than interactive-chat-fast.

## Observation

The main thing I wanted to report separately is that increasing the expert cache budget did not improve performance on this 12 GB configuration.

In this test:

- 10 GB pushed planned GPU memory very close to the device limit.
- 8 GB left more headroom.
- 8 GB was also slightly faster in both prefill and decode.

This may simply be caused by reduced memory pressure and fewer allocation constraints on low-VRAM devices, but it may be useful for the SSD-streaming planner or documentation.

Hope these numbers are useful for low-VRAM CUDA tuning.