NVIDIA / NVIDIA/NemoClaw

[Ubuntu 24.04][Onboard] An interrupted onboard leaves a lock whose owner PID is dead, and onboard --resume gives up after 120 retries instead of reclaiming it

Open
#10,461 2 comments 0 reactions 1 assignee Claimed by @jason-ma-nv View on GitHub
area: onboarding platform: ubuntu
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

Interrupting `nemoclaw onboard` mid-provisioning leaves the host lock directory
`~/.nemoclaw-portable-host.lock` behind, with an `owner` file naming the PID of the process that just died.
The documented recovery command for exactly this situation — `nemoclaw onboard --resume` — then cannot run: it
retries for the lock 120 times and exits non-zero with

```
Error: Failed to acquire lock on /home//.nemoclaw-portable-host.lock after 120 retries
```

Nothing checks whether the recorded owner is still alive. It is not — the PID in the lock's `owner` file is a
dead process. The message offers no remediation either; the user is told only that the retries ran out, not
what holds the lock or how to clear it.

A second, related shape shows up on the same scenario: when the interrupt leaves `~/.nemoclaw/onboard.lock`
instead, resume refuses with "Another NemoClaw onboarding run is already in progress. Lock holder PID: NNNN",
and that PID is also dead. That message at least names the file and suggests removing it, so it is the milder
of the two — but it is the same missing liveness check.

Reproduced 2 out of 2 attempts.

Platform scope: Ubuntu 24.04 x86_64 with an NVIDIA A100-SXM4-40GB. Not attempted elsewhere.
Regression: Unknown — earlier versions not tested.

## Environment

```
Device: x86_64 server, NVIDIA A100-SXM4-40GB
OS: Ubuntu 24.04 LTS
Architecture: x86_64
Docker: 29.7.2
NemoClaw: v0.0.114
OpenShell CLI: 0.0.106
Provider: NVIDIA Endpoints (nvidia/nemotron-3-super-120b-a12b)
```

## Steps to Reproduce

```
1. Start a first-run onboard and let it reach sandbox creation:
NEMOCLAW_YES=1 nemoclaw onboard --name onboard-resume-test --non-interactive --fresh \
--yes-i-accept-third-party-software &
Wait until the output shows "[6/8] Creating sandbox".
2. Interrupt it, as a user would:
pkill -INT -f "nemoclaw onboard --name onboard-resume-test"
(SIGTERM if it does not exit on SIGINT). A partial sandbox is left on the gateway in Error phase.
3. Run the documented recovery command:
nemoclaw onboard --resume --non-interactive --yes-i-accept-third-party-software
4. Inspect the lock and whether its owner still exists:
ls -la ~/.nemoclaw-portable-host.lock
cat ~/.nemoclaw-portable-host.lock/owner
kill -0 $(cat ~/.nemoclaw-portable-host.lock/owner) ; echo "alive=$?"
```

## Expected Result

`onboard --resume` is the recovery path for an interrupted onboard, so it completes. A lock left by a process
that no longer exists is recognised as stale and reclaimed. If it genuinely cannot be reclaimed, the error names
the lock, the owner, and the command to clear it, rather than reporting only that the retry budget ran out.

## Actual Result

Step 3:

```
RESUME_EXIT=1
Error: Failed to acquire lock on /home//.nemoclaw-portable-host.lock after 120 retries
```

Step 4 — the lock is a directory and its owner is dead:

```
ls -la ~/.nemoclaw-portable-host.lock
drwx------ 2 4096 Aug 27 06:40 .
-rw------- 1 7 Aug 27 06:40 owner
-rw------- 1 55 Aug 27 06:40 process-start

cat ~/.nemoclaw-portable-host.lock/owner

kill -0
-> not running (dead)
```

The lock directory's ctime is 06:40:05, i.e. it was created by the onboard that was interrupted at 06:40:27.

On the first of the two runs the lock left behind was `~/.nemoclaw/onboard.lock` instead, and resume reported:

```
Another NemoClaw onboarding run is already in progress.
Lock holder PID:
Started: 2026-08-27T06:36:09.511Z
Wait for it to finish, or remove the stale lock if the previous run crashed:
rm -f "/home//.nemoclaw/onboard.lock"
```

`kill -0 ` also reported the process as dead. Clearing that file by hand let resume proceed to the
next stage, so the lock is the only thing standing between the user and recovery.

Everything else about the interrupted state is sound and worth recording: the partial sandbox is left on the
gateway in Error phase as expected, the lifecycle journal `~/.nemoclaw/onboard-session.json` is written (4018
bytes) and contains no API keys or tokens, and the run never emitted the
"sandbox route reservation ... disappeared while onboarding was in progress" error.

## Logs

```
# run 2 (portable-host lock)
[2] interrupt
interrupted; gateway state:
NAME CREATED PHASE
onboard-resume-test 2026-08-27 06:40:14 Error

[3] lifecycle journal
--- ~/.nemoclaw/onboard-session.json (4018 bytes) secret hits: 0

[4] resume
RESUME_EXIT=1
forbidden-error count (expect 0): 0
Error: Failed to acquire lock on /home//.nemoclaw-portable-host.lock after 120 retries

# run 1 (onboard.lock)
RESUME_EXIT=1
Another NemoClaw onboarding run is already in progress.
Lock holder PID: -> DEAD
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.