[Linux][Onboard] Resume of interrupted managed-vLLM onboarding self-conflicts on GPU memory with its own orphaned vLLM container
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
Resuming an interrupted managed-vLLM onboarding (Ctrl+C after the vLLM container has already launched, then re-running the same onboard command) does not detect or wait on the already-running vLLM container from the interrupted attempt. Instead it starts a brand-new install attempt, which fails on GPU-memory insufficiency because its own orphaned container from the interrupted attempt is still running and holding the GPU.
Related, not a duplicate (different failure mode): #11148 also covers a resume-of-interrupted-managed-vLLM-install scenario, but that report's resume fails on a config conflict (`NEMOCLAW_SERVING_PRESET` vs `NEMOCLAW_VLLM_MODEL`) with the leftover container already `Exited`. This report's resume instead reaches the GPU-availability check and fails there, with the leftover container still `Running` and actively holding GPU memory — a distinct code path and distinct root cause, even though both are triggered by interrupting a managed vLLM install.
Also possibly related (different repro path): a separate report describes onboarding a second, new sandbox while an already-Ready, different sandbox's vLLM server is healthy — that case never attempts the documented reuse path and fails on a port-8000 conflict with the other sandbox's server. This report's repro is same-sandbox self-conflict (the sandbox's own interrupted attempt vs. its own resume attempt), not cross-sandbox reuse.
Platform scope: Reproduced on Linux x86_64 only; other platforms not tested.
Regression: Unknown - earlier versions not tested.
## Environment
```text
Device: Linux x86_64 server, NVIDIA GeForce RTX 5090 D
OS: Ubuntu, Linux 6.17.0-23-generic
Architecture: x86_64
Node.js: v22.23.0
npm: 10.9.8
Docker: 29.7.2
OpenShell CLI: 0.0.116
NemoClaw: v0.0.124
OpenClaw: 2026.7.1
```
## Steps to Reproduce
1. Confirm a compatible managed vLLM profile:
```bash
nemoclaw profiles list
```
2. Start onboarding in the background:
```bash
nemoclaw onboard --profile {compatible-vllm-profile-id} --name recovery-match --yes --non-interactive --yes-i-accept-third-party-software
```
3. Send SIGINT (Ctrl+C) to the onboard process as soon as its log shows the vLLM container has started launching (e.g. a line like "Starting vLLM container" / "Launching vLLM").
4. Confirm onboarding exited without completing, and that the sandbox does not appear as Ready in the inventory:
```bash
nemoclaw list
```
5. Re-run the exact same onboard command from step 2 (the documented resume path).
## Expected Result
Step 5: onboarding resumes and completes without asking the user to edit runtime state — it should detect the already-launched vLLM container from the interrupted attempt and either reuse it or wait for it to finish starting, then bring the sandbox to Ready.
## Actual Result
Step 3-4: onboarding exits with "Onboarding did not finish... nemoclaw onboard --resume" and the sandbox correctly does not appear as falsely Ready. This part matches expected.
Step 5: rerunning the same onboard command does NOT detect or reuse the already-running vLLM container from the interrupted attempt. It instead starts a completely new install attempt, which fails: "vLLM install failed: {model} sets --gpu-memory-utilization=0.7, which requires about 22.3 GiB free on GPU 0, but only 8.7 GiB ... is free." — because its own orphaned container from the interrupted attempt in step 3 is still running and holding the GPU. Confirmed via `docker ps` and `nvidia-smi` that only ONE vLLM container existed throughout this whole sequence (the original one from step 2/3, uptime matched) — so this is unambiguously the resume attempt self-conflicting with its own leftover process from the interrupted attempt, not an unrelated stale container.
## Logs
```text
Step 3 interrupt - onboard log tail right before SIGINT:
Starting vLLM container (nemoclaw-vllm)
Launching vLLM
Launch can take 5 minutes to 30 minutes
After SIGINT:
Onboarding did not finish. Resume from the step that failed with:
nemoclaw onboard --resume --name {sandbox}
Completed steps are skipped; pass --fresh instead to start over.
Step 4 - nemoclaw list does not show recovery-match; docker ps at this point:
nemoclaw-vllm: Up 30 seconds
Step 5 - rerunning the same onboard command:
[3/8] Configuring inference provider
------------------------------------
[non-interactive] Provider: install-vllm
...
Installing vLLM. Progress will print below.
vLLM install failed: NVIDIA Nemotron-3 Nano 4B FP8 sets --gpu-memory-utilization=0.7, which requires about 22.3 GiB free on GPU 0, but only 8.7 GiB of 31.8 GiB is free. Stop other GPU workloads to free at least 13.6 GiB, then resume onboarding.
[non-interactive] Aborting: vLLM install failed. See errors above.
Onboarding did not finish. Resume from the step that failed with:
nemoclaw onboard --resume --name {sandbox}
Completed steps are skipped; pass --fresh instead to start over.
docker ps -a immediately after the failed step 5 - only one vllm container, uptime confirms it is the SAME container from step 2/3:
nemoclaw-vllm: Up 57 seconds
nvidia-smi --query-compute-apps at this point:
pid, used_gpu_memory [MiB], process_name
{pid}, 24226 MiB, VLLM::EngineCore
```
Contributor guide
Research direction
Start by reproducing the interrupted `nemoclaw onboard` flow and inspect the resume path around step [3/8], `install-vllm`, and the `nemoclaw-vllm` container lifecycle. Verify how the existing container is discovered after SIGINT. Done means rerunning the same command reuses or waits for that container and brings the sandbox to Ready without a GPU-memory conflict.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, linux, typescript
- Domain
- ai, cli, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100