NVIDIA / NVIDIA/NemoClaw

[Linux][CLI&UX] read-only status blocks 13 seconds then fails during a rebuild, naming a portable lock

Open
#11,903 0 comments 0 reactions 0 assignees View on GitHub
needs: triage NV QA
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

While an ordinary sandbox rebuild is running, the read-only `status` command cannot be used at all. Every call blocks for about thirteen seconds and then exits 1 with a single line saying it could not acquire a lock after 120 retries. A rebuild can run for tens of minutes, so for that whole window the user has no way to check on the sandbox.

Two things make this worse than a plain wait.

The lock it names is a **portable host lock**, but nothing about the situation is portable. The sandbox was created by ordinary onboarding and the host has never run the portable experimental profile. The lock directory does not exist before the rebuild starts: the ordinary rebuild creates it, holds it, and removes it when it finishes. So the name is not left-over state from an earlier experiment, it is what an ordinary rebuild uses.

The message gives the user nothing to act on. It reports only that a lock could not be acquired after a number of retries. It does not say another operation holds the lock, which operation it is, or that the user should simply wait for the rebuild to finish.

Because each call burns about thirteen seconds before failing, polling status during a rebuild is not practical either — the command itself sets the floor.

This is about a **read-only query** being blocked, not about two mutating commands competing. `status` changes nothing, so it is not obvious why it needs a host-wide lock at all.

- **Platform scope:** Reproduced on DGX Station GB300 (Ubuntu 24.04, aarch64) on v0.0.126 and earlier on an Ubuntu 24.04 x86_64 host on v0.0.123; not architecture-specific. Other platforms not tested.
- **Regression:** No — seen on both v0.0.123 and v0.0.126.
- **OpenShell issue:** No

## Environment

```text
Device: NVIDIA DGX Station GB300
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Node.js: v22.23.2
npm: 10.9.8
Docker: Docker version 29.6.1, build 8900f1d
OpenShell CLI: 0.0.116
NemoClaw: v0.0.126
OpenClaw: 2026.7.1
```

## Steps to Reproduce

Use a host that has never run the portable experimental profile. Confirm that first and record it, because it is the point of the report:

```bash
ls -d ~/.nemoclaw-portable-host.lock
ls -d ~/.config/nemoclaw/portable
ls -d ~/.nemoclaw/hermes-portable-lifecycle
```

None of the three should exist.

1. Onboard a sandbox through ordinary onboarding and confirm it reaches Ready. Do not set any gateway port or gateway state directory override, and do not use any experimental profile:

```bash
nemoclaw onboard --name statuslock --non-interactive --yes
```

2. Measure a baseline: time one status call and record its exit code and output length.

```bash
time nemoclaw statuslock status ; echo "exit=$?"
```

3. In one terminal, start an ordinary rebuild:

```bash
nemoclaw statuslock rebuild --yes
```

4. In a second terminal, while the rebuild is still running, call status repeatedly. Record the exit code, the wall-clock duration, and the full output of each call. Six or more calls.

```bash
time nemoclaw statuslock status ; echo "exit=$?"
```

5. While the rebuild is still running, check whether the portable lock path now exists and what is inside it.

6. After the rebuild finishes, call status once more and compare, and check the portable lock path again.

## Expected Result

The read-only `status` command stays usable while a rebuild is in progress, so the user can see what is happening to the sandbox.

If `status` genuinely has to wait for the rebuild, it says so plainly: that another operation is in progress, which one, and that the user should wait for it to finish. It does not surface a raw lock path and a retry count, and it does not refer to a portable lock on a host with no portable state.

## Actual Result

Baseline before the rebuild, and again after it finishes: about 2.4 seconds, exit 0, and the full status report.

Every status call made during the rebuild blocks and then exits 1 with one line. Four calls across two independent rebuilds:

```text
call 1 exit 1 13.052 s Error: Failed to acquire lock on {home}/.nemoclaw-portable-host.lock after 120 retries
call 2 exit 1 13.078 s Error: Failed to acquire lock on {home}/.nemoclaw-portable-host.lock after 120 retries
call 3 exit 1 13.116 s Error: Failed to acquire lock on {home}/.nemoclaw-portable-host.lock after 120 retries
call 4 exit 1 13.105 s Error: Failed to acquire lock on {home}/.nemoclaw-portable-host.lock after 120 retries
```

That single line is the whole output. No transitional state is ever shown — status goes straight from this error to reporting the sandbox Ready once the rebuild finishes. It never reports a starting state and it never reports the sandbox offline; it simply does not render.

The portable angle, checked directly rather than assumed:

```text
before the rebuild none of the three portable paths exist
during the rebuild the named lock directory exists, created at rebuild start, holding
an owner entry and a process-start entry
after the rebuild the lock directory is gone again
after the whole run none of the three portable paths exist
```

So the ordinary rebuild creates and holds the lock the error names. This is not residue from an earlier portable experiment on this host.

The sandbox itself was fine throughout: the first status call after the rebuild finished returned exit 0 with a healthy full report.

One disclosure about the host: it carries a quarantined portable lock directory from an unrelated experiment two days earlier, at a different path from the one in the error, left untouched for this run. It does not explain the result — the lock in the error message was observed being created by this rebuild — but a host with no portable history at all would close that door completely.

## Related Bugs / not duplicate of

#11052 covers live lock contention between mutating commands — a second onboard, rebuild, or uninstall started while a first one holds the lock — reporting a bare internal error instead of the shipped wait-and-retry guidance. Fixing that message would improve half of what is reported here.

This report is deliberately narrower and different: the blocked command is the read-only `status` query, which mutates nothing. The open question it raises is whether `status` should be taking a host-wide lock at all, which is separate from the wording of the contention message.

There is also existing coverage of a lock left behind by an interrupted run whose owner process is already dead. That is not this: here the owner is alive and the rebuild is legitimately in progress.

## Logs

```text
Not captured beyond the terminal output quoted in Actual Result.
```

Contributor guide

Open the contributing guide

Research direction

Start by tracing the CLI entry points for `status` and `rebuild`, then reproduce the concurrent commands from the issue while observing the shared host-lock behavior. Done means status remains usable during a rebuild, or reports the active operation and an actionable wait message without the misleading portable-lock detail.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.