[All Platforms][Inference] Custom NEMOCLAW_VLLM_PORT: destroy leaks ~180GB GPU memory and status probes wrong port
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
Onboarding a sandbox with a custom `NEMOCLAW_VLLM_PORT` leaves the standalone managed vLLM Docker container running after `nemoclaw {sandbox} destroy`, permanently holding its full GPU memory reservation and blocking subsequent onboarding attempts on the same GPU. A second, related defect found in the same test pass: `nemoclaw {sandbox} status` reports the vLLM backend as unreachable when a custom port is used, because it probes the hardcoded default port 8000 instead of the actual configured port.
Platform scope: All Platforms -- reproduced on DGX Station (aarch64), but root cause is the status/destroy CLI path (hardcoded port check, missing container cleanup) with no platform/hardware dependency; expected to reproduce identically on any platform.
Regression: Unknown — earlier versions not tested.
## Environment
```text
Device: DGX Station, 2x GPU (RTX PRO 6000 Blackwell Max-Q 97887 MB, GB300 256703 MB)
OS: Ubuntu, Linux 6.17.0-1029-nvidia-64k
Architecture: aarch64
Docker: 29.6.1
OpenShell CLI: 0.0.106
NemoClaw: v0.0.121
OpenClaw: 2026.7.1 (in the destroyed sandbox)
```
## Steps to Reproduce
1. Pick a free canonical TCP port:
```bash
VLLM_PORT=$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1",0)); print(s.getsockname()[1]); s.close()')
```
2. Onboard with the custom port against a compatible managed vLLM profile:
```bash
NEMOCLAW_VLLM_PORT="${VLLM_PORT}" nemoclaw onboard --profile vllm.dgx-station-gb300.single.nemotron-3-nano-4b-fp8 --name vllm-custom-port --yes --non-interactive --yes-i-accept-third-party-software
```
3. Once Ready, run `nemoclaw vllm-custom-port status` and note the "Inference (vllm backend)" line.
4. Run `nemoclaw vllm-custom-port destroy --yes`.
5. Run `docker ps -a` and `nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv` to inspect what remains.
6. Attempt a second onboard on the same host using the same profile:
```bash
nemoclaw onboard --profile vllm.dgx-station-gb300.single.nemotron-3-nano-4b-fp8 --name vllm-default-port --yes --non-interactive --yes-i-accept-third-party-software
```
## Expected Result
- Step 3: status reports the vLLM backend healthy/reachable on the actually-configured custom port.
- Steps 4-5: destroy removes the sandbox AND its managed vLLM container/listener; `docker ps -a` shows no leftover vLLM container and GPU memory is fully released.
- Step 6: a subsequent onboard on the same host succeeds since GPU memory was reclaimed by step 4.
## Actual Result
**Step 3** — `nemoclaw vllm-custom-port status` reports:
```text
Inference (vllm backend): unreachable (http://127.0.0.1:8000/v1/models)
Local vLLM is selected for inference, but the host probe to http://127.0.0.1:8000/v1/models failed. Start the local vLLM server and retry. (curl failed (exit 7): curl: (7) Failed to connect to 127.0.0.1 port 8000 after 0 ms: Couldn't connect to server)
```
This is despite the listener being confirmed live via `ss -ltnp` on the actual configured port (46145 in this run, not 8000). `docker ps` at this point shows the container correctly mapped `0.0.0.0:46145` to container port `8000/tcp`. The status check is probing the wrong host port — it should check the configured custom port, not a hardcoded 8000.
**Steps 4-5** — `nemoclaw vllm-custom-port destroy --yes` reports `Sandbox 'vllm-custom-port' destroyed` (exit 0) and removes the sandbox's own Docker image, but the separate standalone `nemoclaw-vllm` container is left running:
```text
$ docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
812942a9d59f "/bin/bash -lc 'expo..." 11 minutes ago Up 11 minutes 0.0.0.0:46145->8000/tcp, [::]:46145->8000/tcp nemoclaw-vllm
$ nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csv
pid, process_name, used_gpu_memory [MiB]
361062, VLLM::EngineCore, 179974 MiB
```
That matches vLLM's `--gpu-memory-utilization=0.7` reservation for that GPU — i.e. destroy left ~70% of the GPU (180 GB) permanently allocated to a container the CLI itself just reported as removed.
**Step 6** — the follow-up onboard attempt fails at the vLLM install step:
```text
vLLM install failed: NVIDIA Nemotron-3 Nano 4B FP8 sets --gpu-memory-utilization=0.7, which requires about 175.5 GiB free on GPU 1, but only 74 GiB of 250.7 GiB is free. Stop other GPU workloads to free at least 101.4 GiB, then resume onboarding.
```
This is caused directly by the leaked container from steps 4-5. Manually running `docker rm -f nemoclaw-vllm` on the host immediately freed the GPU back to ~255 GB free, confirming the leftover container was the sole cause.
## Logs
```text
$ nemoclaw vllm-custom-port destroy --yes
Deleting sandbox 'vllm-custom-port'...
[services] cloudflared was not running
[services] Stopping in-sandbox OpenClaw gateway (sandbox: vllm-custom-port)...
[services] OpenClaw gateway was not running inside sandbox.
[services] cloudflared was not running
[services] All services stopped.
Untagged: nemoclaw-sandbox-local:vllm-custom-port-
Deleted: sha256:
Removed Docker image nemoclaw-sandbox-local:vllm-custom-port-
Shared NemoClaw gateway preserved. Re-run 'openshell gateway remove nemoclaw' to remove it,
or pass '--cleanup-gateway' / set NEMOCLAW_CLEANUP_GATEWAY=1 next time. (#2166)
Sandbox 'vllm-custom-port' destroyed
```
Contributor guide
Assessment
This issue has not been assessed yet.