DGX Spark managed vLLM: MTP cold-start NV_ERR_NO_MEMORY, negative memory estimate, and long-context agent instability

## Summary

On a real DGX Spark/GB10, the current NemoClaw `v0.0.86` managed-vLLM profile for `nvidia/Qwen3.6-35B-A3B-NVFP4` shows a reproducible MTP-specific memory-planning risk and long-context agent instability:

1. During the first managed-default cold start, the NVIDIA kernel driver logged a real `NV_ERR_NO_MEMORY` from `_memdescAllocInternal` at the exact moment vLLM started loading the MTP drafter model, even though Linux still reported roughly 53.5 GiB `MemAvailable` and swap was unused.
2. With MTP enabled, vLLM later reported an impossible negative CUDA-graph memory estimate (`-20.19 GiB`) and selected 42.04 GiB of KV cache.
3. Removing only MTP changed the CUDA-graph estimate to a sane `+0.54 GiB`, reduced KV allocation to about 23 GiB, increased observed host headroom under load from ~41 GiB to ~62 GiB, eliminated NVIDIA/kernel faults in the test matrix, and passed a 20-turn real-agent + 128K direct endurance run.
4. Disabling async scheduling alone did not fix the high-context agent timeout. Disabling both MTP and async was also worse than MTP-off/async-on for the same near-timeout replay.

The complete host freeze described below was **not** reproduced in the controlled matrix, so this issue does not claim that MTP is proven to be the fatal-freeze root cause. It does provide concrete evidence that the managed MTP configuration can reach a low-level NVIDIA allocation failure and incorrect memory planning on Spark.

## Original field failure

The earlier message trace clarifies that this was a sequence of related hard freezes, not one configuration from the beginning:

1. The reporter first served **Nemotron-3-Super** through a manually managed vLLM setup on DGX Spark. The host hard-froze after a few agent messages.
2. Assuming model size/OOM was the cause, they switched to **Qwen3.6-35B** with `--max-model-len 131072`. The same full-host freeze occurred, only later in the workload.
3. Their manual Qwen configuration was subsequently identified explicitly as `nvidia/Qwen3.6-35B-A3B-NVFP4` in `nvcr.io/nvidia/vllm:26.05.post1-py3`, with 0.35 GPU memory utilization, 196608 max context, two sequences, FlashInfer, FP8 KV cache, async scheduling, prefix caching, and MTP with three speculative tokens.
4. They then moved to NemoClaw's **managed vLLM** path and used the defaults selected for DGX Spark. The later report—“crashed again on managed vLLM (same model, same host)”—refers to Qwen on that managed path. This later managed-Qwen failure is the direct target of the controlled `v0.0.86` experiments below.

The reporter's original environment was NemoClaw `v0.0.73`, not `v0.0.86`. Their sandbox also ran four MCP servers (three Go binaries and one Python process). Two MCP servers used external network policies; two were local/SQLite-backed.

The workload most strongly correlated with freezes was an hourly cron job processing multiple open tickets. For each ticket it could read state, search email, evaluate replies, check a calendar/external API, send follow-up mail, and update database/status records. This produced many sequential tool calls and loops over multiple tickets. A lightweight cron job that only checked for new file attachments did not freeze the host.

The reporter also observed five consecutive `tool_search_code` timeouts before a sixth attempt succeeded and raised the possibility that isolated Node subprocesses from timed-out calls might not be cleaned up. No subprocess leak was proven, so this remains an unverified possible cofactor rather than part of the MTP conclusion.

In every fatal occurrence, the host-level symptom was the same: SSH and ping died, the local monitor stopped, and no final software-side error was persisted. The machine required a physical restart. After a freeze, the sandbox could also be left unusable and require recreation.

The reporter had seen `NVRM NV_ERR_NO_MEMORY` events hours before some fatal freezes but not at the recorded crash time. They initially suspected page-cache growth/unified-memory accounting and considered cache-dropping and `vm.vfs_cache_pressure=200`; those were hypotheses/workarounds from the field thread, not validated fixes. Docker memory limits were also reported as ineffective for unified GPU allocations.

Last complete five-second monitor sample before that failure:

```text
Mem: 121 GiB total, 80 GiB used, 15 GiB free, 27 GiB buff/cache, 41 GiB available
Swap: 143 GiB total, 0 B used

PSI memory:
some avg10=0.00 avg60=0.00 avg300=0.00
full avg10=0.00 avg60=0.00 avg300=0.00

docker stats:
sandbox 58.95% CPU 612 MiB
nemoclaw-vllm 107.83% CPU 30.27 GiB
```

Five seconds later memory and PSI were still essentially unchanged, vLLM CPU was 79.95%, and the next sample stopped partway through `docker stats` as the host became unreachable.

That did not resemble a gradual Linux host OOM: memory showed no upward trend, PSI remained flat, swap was unused, and ~41 GiB was reported available. Because a five-second sampler can miss a sub-second unified-memory allocation and because GPU/DMA allocation constraints are not represented fully by container RSS or Linux `MemAvailable`, a driver/UVM allocation or GPU-kernel failure remained plausible.

## Why file this in NemoClaw

The initial Nemotron and Qwen failures happened with manually managed vLLM, so they show that the underlying Spark/unified-memory failure is not necessarily NemoClaw-specific. This issue focuses on the later failure after the reporter switched to the managed path and on the concrete risk reproduced with the current managed Spark recipe. NemoClaw launches Qwen with MTP speculative decoding, async scheduling, 262K context, four sequences, 8192 batched tokens, FlashInfer attention, FP8 KV cache, chunked prefill, and prefix caching together.

The broader validation work in #5615 calls for a real local-assistant workflow and a settings matrix on DGX Spark. This issue supplies that physical-hardware matrix and isolates one actionable default: MTP/speculative decoding.

I searched open and closed NemoClaw issues for `NV_ERR_NO_MEMORY`, negative CUDA-graph memory, the exact Qwen model, managed-vLLM freezes, and MTP. I did not find an issue covering this exact evidence and mitigation.

## Environment

The table below is the **controlled reproduction environment**, not the reporter's original `v0.0.73` manual-vLLM environment.

| Component | Value |
| --- | --- |
| Hardware | DGX Spark / NVIDIA GB10, 121 GiB usable unified memory |
| DGX OS | build 7.4.0, OTA level 7.5.0 |
| Kernel | `6.17.0-1026-nvidia` |
| NVIDIA driver | `580.159.03` |
| NemoClaw | tag `v0.0.86`, commit `e7e8b674e0d7fcaa37ad3806d5f39df352e702c4` |
| OpenShell | `0.0.85` |
| OpenClaw | `2026.6.10` |
| Model | `nvidia/Qwen3.6-35B-A3B-NVFP4` |
| Managed image | `sha256:9204569b17ee4c0eff75194b8e6e458479c8aee18953b5ab9cf359fcdac659e2` |
| vLLM | `0.21.0+2325b6f0.dev` |
| FlashInfer | `0.6.11.post3+0e4b4948.nvinternal.26.05.post1.devel.cu132.51679275` |
| PyTorch | `2.12.0a0+5aff3928d8.nv26.05` |
| Docker | `29.2.1` |

NemoClaw status after installation:

```text
Sandbox: repro-spark
Model: nvidia/Qwen3.6-35B-A3B-NVFP4
Provider: vllm-local
Inference: healthy
Inference (vllm backend): healthy
Host GPU: yes
Sandbox GPU: enabled (CUDA verified)
```

## Exact managed command

The baseline container command from `docker inspect nemoclaw-vllm` was:

```bash
pip install vllm[fastsafetensors] && \
vllm serve nvidia/Qwen3.6-35B-A3B-NVFP4 \
--tensor-parallel-size 1 \
--pipeline-parallel-size 1 \
--data-parallel-size 1 \
--port 8000 \
--trust-remote-code \
--max-model-len 262144 \
--gpu-memory-utilization 0.4 \
--dtype auto \
--quantization modelopt \
--kv-cache-dtype fp8 \
--attention-backend flashinfer \
--moe-backend marlin \
--max-num-seqs 4 \
--max-num-batched-tokens 8192 \
--enable-chunked-prefill \
--async-scheduling \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--reasoning-parser qwen3 \
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' \
--load-format fastsafetensors
```

Container settings included `--gpus all`, host IPC, port 8000, the Hugging Face cache bind-mounted at `/root/.cache/huggingface`, restart policy `unless-stopped`, and label `com.nvidia.nemoclaw.managed-vllm=true`.

## Critical NVIDIA driver evidence

During the first exact managed-default startup, the kernel logged:

```text
Jul 17 14:03:21.012107 spark-0f7a kernel: NVRM: nvCheckOkFailedNoLog:
Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from
_memdescAllocInternal(pMemDesc) @ mem_desc.c:1359
```

The vLLM log at the same instant was entering the MTP-specific path:

```text
21:03:17 Loading weights took 13.79 seconds
21:03:21 Loading drafter model...
21:03:21 Asynchronous scheduling is enabled.
21:03:21 Using TRITON Unquantized MoE backend ...
21:03:21 Loading safetensors ... [MTP drafter]
21:03:25 Detected MTP model. Sharing target model embedding weights with the draft model.
21:03:25 Detected MTP model. Sharing target model lm_head weights with the draft model.
21:03:26 Model loading took 21.94 GiB memory
```

Nearest host sample after the driver error:

```text
MemAvailable: 54,237,844 kB (~53.5 GiB)
Swap used: 4 kB
docker stats nemoclaw-vllm: 24.69 GiB
PSI before the allocation: some/full avg10=0.00
```

The host and server survived this occurrence. It is therefore a recoverable driver allocation failure in this run, not the fatal host freeze. It is still important because it proves that the managed profile can fail a low-level NVIDIA allocation while general-purpose Linux memory appears abundant.

Later in the same startup, vLLM reported:

```text
Estimated CUDA graph memory: -20.19 GiB total
Available KV cache memory: 42.04 GiB
GPU KV cache size: 3,548,441 tokens
Maximum concurrency for 262,144 tokens per request: 13.54x
```

The negative estimate is included as supporting evidence/side effect, not asserted as the direct cause of the driver error. The driver error occurred while loading the drafter, before graph profiling printed the negative estimate.

The same cached baseline container was later restarted successfully without another `NV_ERR_NO_MEMORY`; the driver event was observed once on the initial cold startup.

## Controlled matrix

Only one configuration dimension was changed at a time where possible:

| Run | Async scheduling | MTP | Max context | Max sequences | Max batched tokens | Purpose |
| --- | --- | --- | ---: | ---: | ---: | --- |
| A | on | 3 draft tokens | 262144 | 4 | 8192 | Exact managed default |
| B | off | 3 draft tokens | 262144 | 4 | 8192 | Test async scheduling independently |
| C | on | off | 262144 | 4 | 8192 | Test MTP independently; balanced candidate |
| D | off | off | 262144 | 4 | 8192 | Disable both |
| E | off | off | 32768 | 1 | 4096 | Conservative bounded control |

Every profile kept the same host, image digest, model files, mount, 0.4 GPU memory utilization, ModelOpt quantization, FlashInfer attention, FP8 KV cache, Marlin MoE backend, chunked prefill, prefix caching, auto tool choice, `qwen3_coder` tool parser, and `qwen3` reasoning parser.

Run C removed only `--speculative-config`. Run B replaced async scheduling with `--no-async-scheduling`. Run D combined those changes. Run E also reduced context, active sequences, and batched tokens.

## Startup memory comparison

| Profile | Loaded model memory | CUDA-graph estimate | Available KV cache | Observed host headroom under load | Driver/kernel result |
| --- | ---: | ---: | ---: | ---: | --- |
| A/B: MTP on | 21.94 GiB | `-20.19 GiB` | 42.04 GiB | ~41 GiB | One Run A cold-start `NV_ERR_NO_MEMORY` |
| C: MTP off, async on | 20.37 GiB | `+0.54 GiB` | 23.29 GiB | ~62 GiB | Clean |
| D: MTP off, async off | 20.37 GiB | `+0.53 GiB` | 22.99 GiB | ~63 GiB | Clean |
| E: bounded control | 20.37 GiB | `+0.52 GiB` | 23.47 GiB | ~63 GiB | Clean |

Removing MTP saves about 1.57 GiB of loaded model memory directly, but changes the memory planner's KV decision by roughly 19 GiB. That secondary allocation difference is much larger than the drafter weight difference.

Reducing maximum context to 32K did not materially reduce total KV allocation: vLLM used the available capacity for a larger theoretical number of shorter contexts. The bounded profile improves containment by limiting each request and serializing active sequences, not by returning much additional reserved capacity.

## Workloads

Two independent classes of workload were used.

### Direct OpenAI-compatible load

A dependency-free deterministic client grew tool-call/tool-result conversations through `/v1/chat/completions`. Each turn:

1. Asked the model to call one declared synthetic tool.
2. Stored the assistant tool call in conversation history.
3. Added a deterministic ~4 KiB tool result.
4. Recorded full response, prompt-token usage, completion usage, finish reason, and latency.
5. Flushed and `fsync`'d the JSONL record before the next request.

Runs covered one and two sessions, 28K, 34K, 65K, 128K, and 132K prompt contexts, deterministic seeds, and mixed operation alongside real agents.

### Real OpenClaw agent load

The agent workload ran through NemoClaw/OpenShell, not directly against vLLM. It repeatedly used real file and shell tools to:

- Create CSV and JSONL datasets.
- Inspect heads/tails and selected records.
- Compute sums and parity counts independently.
- Audit malformed rows and inconsistent numeric results.
- Edit reports and manifests.
- Calculate line counts, sizes, and SHA-256 digests.
- Re-read saved files and verify claims before replying.

Tests included:

- An initial 20-turn real tool soak.
- Reuse of the problematic ~138K--141K agent history.
- Multiple fresh bounded agent sessions.
- Three agents plus direct inference concurrently.
- One growing 20-turn agent history alongside a 128K direct stream.

Each agent turn had a 360-second client timeout so server survival and request failure could be classified separately.

## Aggregate results

Across the matrix:

- **760 direct requests completed; 0 direct request errors.**
- **78 real agent workflows completed.**
- **3 agent workflows hit the six-minute timeout:** the aggressive managed-default continuation, the Run B high-context replay, and the Run D high-context replay.
- **0 vLLM process crashes.**
- **0 container restarts during workload.**
- **0 host/SSH losses in the controlled tests.**
- **0 complete host freezes reproduced.**

## Detailed results

| Run/workload | Direct result | Latency | Agent result |
| --- | --- | --- | --- |
| A direct single | 34,271 tokens, 23 requests, 0 errors | median 1.73s, p95 2.03s, max 2.10s | N/A |
| A direct concurrent | 65,820 / 65,774 tokens, 86 requests, 0 errors | median 3.38s, p95 5.06s, max 9.06s | N/A |
| A aggressive mixed | 131,324 / 132,191 tokens, 171 requests, 0 errors | median 5.76s, p95 55.62s, max 57.41s; 18 >30s | Existing ~140K agent history timed out after malformed tool-call retries |
| B: async off, MTP on | 65,891 / 65,719 tokens, 85 requests, 0 errors | median 4.89s, p95 56.19s, max 60.52s; 18 >30s | Same high-context replay timed out at 361s |
| C: MTP off, async on | 65,891 / 65,704 tokens, 86 requests, 0 errors | median 3.79s, p95 53.83s, max 55.23s; 10 >30s | Same high-context replay completed in 320s |
| D: both off | 65,891 / 65,774 tokens, 86 requests, 0 errors | median 4.04s, p95 54.63s, max 57.12s; 12 >30s | Same high-context replay timed out at 361s |
| E direct 32K control | 29,508 / 28,185 tokens, 39 requests, 0 errors | median 4.16s, p95 4.69s, max 4.78s | N/A |
| E mixed | 28,185 / 28,110 tokens, 38 requests, 0 errors | median 4.35s, p95 5.97s, max 9.36s | 6/6 workflows passed |
| E four-client repeat | 28,110-token direct stream, 19 requests, 0 errors | median 10.67s, p95 15.04s, max 16.63s | Three agents passed 15/15 workflows; longest five-turn agent wall time 341s |
| C four-client fresh repeat | 66,388-token direct stream, 44 requests, 0 errors | median 3.61s, p95 4.48s, max 4.93s | Three agents passed 15/15 workflows |
| C endurance | 128,171-token direct stream, 83 requests, 0 errors | median 3.30s, p95 4.27s, max 4.83s | One growing session passed 20/20 turns |

One Run E turn reported an internal failed `xxd -> show -> od` sub-tool chain, recovered by recomputing the values through other tools, verified the result, and completed successfully. No unhandled parser or request failure occurred in the Run C fresh-repeat or endurance agent workflows.

## High-context agent failure behavior

The managed-default agent session file reached roughly 138K--141K input tokens. At that point, the model repeatedly emitted invalid tool-call payloads, including arguments represented as a string instead of the required object, missing required IDs, and malformed keys such as `command>`.

OpenClaw correctly rejected those tool calls. Validation/retry cycles then generated more long model turns, with roughly 58--70-second gaps between attempts. Concurrent direct requests remained alive but showed 45--60-second tail latency. The client eventually timed out and the server-side request was aborted. vLLM and the host stayed healthy.

This separates two interacting concerns:

1. MTP-specific memory planning and low-level allocation risk.
2. Model/tool-parser reliability at very large agent histories, which creates retry loops and contention without necessarily crashing vLLM.

## Timestamp-windowed host telemetry

Raw monitors sampled `free -b`, `/proc/pressure/memory`, `vmstat`, selected `/proc/meminfo`, Docker stats, `nvidia-smi`, `/health`, the live kernel journal, and vLLM logs. Host statistics below were recalculated only inside each workload's recorded start/end timestamps.

| Profile | Minimum `MemAvailable` | Maximum memory used | Maximum swap used | Max PSI full avg10 | Kernel/GPU faults in workload |
| --- | ---: | ---: | ---: | ---: | ---: |
| A aggressive | 41.11 GiB | 80.58 GiB | effectively 0 | 0.00 | 0 during workload; the cold-start driver OOM occurred earlier in Run A |
| B | 40.70 GiB | 80.99 GiB | effectively 0 | 0.39 | 0 |
| C original replay | 61.90 GiB | 59.79 GiB | effectively 0 | 0.00 | 0 |
| C fresh repeat | 62.12 GiB | 59.57 GiB | effectively 0 | 0.00 | 0 |
| C endurance | 62.19 GiB | 59.50 GiB | effectively 0 | 0.00 | 0 |
| D | 63.18 GiB | 58.51 GiB | effectively 0 | 0.00 | 0 |
| E | 62.90 GiB | 58.79 GiB | effectively 0 | 0.00 | 0 |

The ~80 GiB used / ~41 GiB available pattern from the original report occurred during healthy, responsive managed-MTP operation in this matrix. It is not sufficient evidence of gradual host OOM. It is also clearly associated here with the MTP-enabled memory plan; MTP-off workloads retained about 21 GiB more host headroom.

## Interpretation

### What the evidence supports

- MTP is required for the observed cold-start drafter allocation path.
- MTP correlates with one real NVIDIA `NV_ERR_NO_MEMORY` event on cold start.
- MTP correlates with the impossible negative CUDA-graph estimate.
- MTP changes vLLM's KV allocation by roughly 19 GiB and reduces measured host headroom by roughly 21 GiB.
- Removing MTP produced clean startup planning and passed the strongest endurance workload.
- Async scheduling alone is not the cause of the reproduced high-context failure: Run B still timed out.
- With MTP removed, keeping async scheduling enabled gave the best throughput and was the only B/C/D high-context replay to finish before the six-minute timeout.

### What the evidence does not prove

- The original fatal host freeze was not reproduced, so MTP is not proven to be its sole cause.
- The negative CUDA-graph estimate printed after the driver event and is not established as its direct trigger.
- General Linux free memory cannot rule out a sub-second, contiguous, DMA, address-space, or other driver-specific allocation failure on GB10 unified memory.
- Run C's 320-second completion versus B/D's 361-second timeout is near the configured boundary and should not be treated as a large statistical margin by itself.

## Recommended NemoClaw mitigation

For the balanced managed Spark default, remove this argument:

```text
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'
```

Keep async scheduling enabled initially. This is the tested Run C profile.

For users who prioritize containment over throughput, document an optional conservative profile:

```text
--no-async-scheduling
--max-model-len 32768
--max-num-seqs 1
--max-num-batched-tokens 4096
```

That profile passed all tests but intentionally serialized requests: under four-client load, direct median latency increased from roughly 4 seconds to 10.67 seconds, and one five-turn agent took 5m41s.

## Suggested acceptance criteria

- [ ] Remove or gate MTP for the managed DGX Spark Qwen3.6 profile.
- [ ] Add a physical-Spark cold-start test that fails on `NVRM`, `Xid`, `GSP`, `UVM`, or `NV_ERR` kernel events, not only an unhealthy `/v1/models` endpoint.
- [ ] Add a real agent/tool workflow with growing conversation history.
- [ ] Include at least two concurrent streams and mixed prefill/decode transitions.
- [ ] Record CUDA-graph estimate, loaded model memory, KV-cache choice, and host `MemAvailable` as test artifacts.
- [ ] Run a 30--60-minute soak and verify host/SSH survival.
- [ ] Escalate the cold-start `NV_ERR_NO_MEMORY` plus the negative memory estimate to the vLLM/NVIDIA container owners with the exact image digest and command.

## Reproduction outline

1. On a DGX Spark matching the environment above, install NemoClaw `v0.0.86` and select the managed local-vLLM provider/model.
2. Before starting the container, stream the kernel journal and capture host memory/PSI once per second from another machine if possible.
3. Start the exact managed command and watch the point where vLLM logs `Loading drafter model...`.
4. Record any `NV_ERR_NO_MEMORY` and the later CUDA-graph/KV-cache estimates.
5. Run a deterministic direct tool-call conversation to at least 65K, then two concurrent sessions.
6. Run a persistent real OpenClaw tool workflow so one agent history grows substantially.
7. Repeat with only `--speculative-config` removed.
8. Compare startup memory planning, host headroom, kernel events, request tails, and agent completion.

After a hard freeze and forced reboot, collect before restarting vLLM:

```bash
sudo journalctl -b -1 -k -o short-monotonic --no-pager
sudo journalctl -b -1 -p warning..alert --no-pager
sudo find /sys/fs/pstore -maxdepth 1 -type f -print -exec sed -n '1,300p' {} \;
sudo nvidia-bug-report.sh
```

## Follow-up: official Nemotron-3-Super NVFP4 on the same Spark

Because the original field sequence began with a manually served
Nemotron-3-Super freeze, I also tested the currently published Spark-capable
checkpoint, `nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4`, on the same host.
The original trace did not contain the exact Nemotron checkpoint,
quantization, image, or serve command, so this is a current best-supported
reproduction rather than a byte-identical recreation.

### Pinned Nemotron environment

| Component | Value |
| --- | --- |
| Checkpoint revision | `4f0cf9daaeb7a4d5e23f80a00e7ed15f0e03caf6` |
| Image | `vllm/vllm-openai:v0.20.0` |
| Image digest | `sha256:04563c302537a91aa49ebdfbceda96111c5712275999b7e8804fa598f0b5641d` |
| Checkpoint size | 17 shards / 74.80 GiB reported by vLLM |
| Integrity check | all 165,860 indexed tensors present; zero missing/unexpected; no incomplete shards |

Xet download was unreliable on this host: high-performance Xet emitted repeated
HTTP 416 reconstruction errors and duplicated transfer accounting. Disabling
Xet (`HF_HUB_DISABLE_XET=1`) completed over ordinary HTTP at roughly
38.5--40.5 MiB/s. This was a download-path finding, not a runtime freeze.

### Nemotron matrix

| Run | GPU util. | Context | Sequences | MTP | Startup/kernel result | Workload result |
| --- | ---: | ---: | ---: | --- | --- | --- |
| N2 | 0.70 | 32,768 | 1 | off | One recoverable `NV_ERR_NO_MEMORY`; server healthy | 20/20 growing direct turns; 5/5 real-agent turns |
| N3 | 0.70 | 32,768 | 1 | 3 tokens | No new driver error | 19/19 growing direct turns; 5/5 real-agent turns |
| N4 | 0.90 | 1,000,000 | 4 | 3 tokens | **Two recoverable `NV_ERR_NO_MEMORY` events**; server healthy | 144K prompt, 4x concurrent 68K--75K prompts, and 5/5 real-agent turns passed |

N4 follows NVIDIA's published vLLM-on-DGX-Spark profile, including
`VLLM_ALLOW_LONG_MAX_MODEL_LEN=1`, vLLM 0.20.0, FP8 KV cache, Marlin, async
scheduling, four sequences, 0.90 memory utilization, and three MTP draft
tokens. The revision pin, ordinary-HTTP download, and explicit 4096 scheduled
token bound were added for reproducibility.

The first 1M preflight omitted `VLLM_ALLOW_LONG_MAX_MODEL_LEN=1` and failed
safely before model load because the checkpoint config's native position limit
is 262,144. The official model card explicitly documents the override; after it
was added, N4 loaded. This preflight was not an OOM or freeze.

### Nemotron startup/memory comparison

| Metric | N2 MTP off | N3 MTP on | N4 official pressure |
| --- | ---: | ---: | ---: |
| Reported model load | 69.54 GiB | 75.03 GiB | 75.03 GiB |
| Drafter second pass | none | 71.87s | 77.87s |
| CUDA-graph estimate | +0.54 GiB | +0.83 GiB | +0.29 GiB |
| CUDA-graph actual | 0.52 GiB | 0.45 GiB | 0.35 GiB |
| GPU KV capacity | 502,656 tokens | 195,120 tokens | 1,175,056 tokens |
| Minimum observed host headroom | ~31 GiB | ~29 GiB | **3.22 GiB** |
| Peak observed swap | ~0.9 GiB | ~1.8 GiB | 2.16 GiB |
| New NVIDIA allocation errors | 1 | 0 | **2** |

At N4 startup, the kernel logged twice:

```text
NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory
[NV_ERR_NO_MEMORY] (0x00000051) returned from
_memdescAllocInternal(pMemDesc) @ mem_desc.c:1359
```

vLLM then continued and reported:

```text
Estimated CUDA graph memory: 0.29 GiB total
GPU KV cache size: 1,175,056 tokens
Maximum concurrency for 1,000,000 tokens per request: 6.49x
CUDA graph pool memory: 0.35 GiB actual, 0.29 GiB estimated
Application startup complete
```

This is an important side result: unlike managed Qwen, Nemotron's graph
estimate was positive. Therefore the negative Qwen CUDA-graph estimate is not
required for the NVIDIA driver allocation failure. High unified-memory
pressure during initialization is the common signal.

### N4 high-pressure workloads

- One real 144,694-token prompt completed in 111.04s with a valid tool call.
- Four simultaneous prompts of 68,403--74,588 tokens completed in
91.41--172.03s with zero HTTP/transport errors. Three emitted valid tool
calls; one consumed its 128-token reasoning budget before the call.
- The real NemoClaw/OpenClaw filesystem workflow passed 5/5 turns in 263s,
71s, 92s, 48s, and 48s.
- Across 70 speculative-decoding metric windows, 7,378 of 10,308 draft tokens
were accepted (71.58%).
- Minimum sampled `MemAvailable` was 3.22 GiB; peak swap use was 2.16 GiB.
- The container had zero restarts, `OOMKilled=false`, and HTTP health remained
200 until it was intentionally stopped.
- No new driver allocation error appeared during inference; the two N4 events
were startup/cache-initialization events.

The full-host freeze still did **not** reproduce. These results strengthen the
issue by reproducing the same low-level precursor with the original model
family and official high-pressure Spark settings, but they do not prove MTP or
any one flag is the fatal-freeze root cause. N2, with MTP off, also logged one
recoverable allocation event, and N4 deliberately changed utilization,
context, and concurrency together to match the published profile.

Recommended next isolation matrix: cold-boot repetitions at 0.90/262K with MTP
on versus off, followed by 1M, while a second host persists the kernel journal
and heartbeat. That separates MTP, context, and utilization effects and avoids
losing the final sample if the Spark hard-freezes.

## Limitations

- One DGX Spark, one OS/driver stack, one image, and one test day.
- The fatal full-host freeze did not occur in the controlled matrix.
- Monitoring was local in this experiment rather than persisted by a second host.
- The B/C/D ~140K tests reused a mutable OpenClaw session, so each replay included the preceding success/abort in history and was not byte-identical.
- The 360-second agent timeout made the C versus B/D high-context distinction narrow.
- Run E cannot exercise 140K because the profile intentionally caps context at 32K.
- The cold-start driver error occurred once; later cached baseline restarts were clean.
- The Nemotron follow-up also did not reproduce a fatal freeze. Its exact
original field command is unknown, and N4's strongest profile jointly changes
memory utilization, context, sequences, and MTP to match the official recipe.
- Early monitor child processes continued after the terminal wrapper received Ctrl-C. Request/agent artifacts were unaffected; host metrics above were recomputed from workload timestamp windows, and the orphan monitors were terminated.

## Checklist

- [x] Tested on physical DGX Spark/GB10 hardware.
- [x] Used the exact latest tagged NemoClaw managed profile before changing one dimension at a time.
- [x] Included direct inference and real OpenClaw tool workloads.
- [x] Included long context, concurrency, fresh sessions, and a growing-session endurance run.
- [x] Preserved exact image/version/command evidence.
- [x] Searched existing NemoClaw issues; no exact duplicate found.
- [x] Restored the supported managed-default container after testing; it is healthy.

Related: #5615