NVIDIA / NVIDIA/NemoClaw

[Ubuntu 24.04][Upgrade] an interrupted rebuild cannot be resumed after an in-place upgrade; the retained recovery observation is gone and the journal is stranded

Open
#11,975 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

A rebuild that is interrupted before it finishes leaves a resumable journal, as intended. After an in-place upgrade to the next release, that journal can no longer be resumed:

```text
The retained rebuild MCP recovery observation is unavailable, so rebuild cannot safely
resume. No sandbox deletion was attempted.
```

The rebuild exits 1 and there is no way to continue the interrupted operation. The journal files themselves survive the upgrade — they are still on disk, and their count is unchanged before and after — so what is lost is the recovery observation the resume path needs, not the journal.

The safety half of the behaviour is correct and worth keeping: the command refuses rather than guessing, states plainly that no sandbox deletion was attempted, and the sandbox is still registered and intact afterwards. Nothing is destroyed. What is missing is any way forward: resuming fails, and the message names no command, no state to repair, and no alternative path.

This matters because interruption plus upgrade is a realistic sequence — a rebuild is interrupted, the operator upgrades, and then expects to pick the rebuild back up.

Secondary observation from the same run, already covered by an existing report (#10461): the interrupted rebuild leaves an onboard lock whose owner process is dead, and that lock blocks every later command including `list`, until it is removed by hand.

- **Platform scope:** Reproduced on Ubuntu 24.04 x86_64 (CPU-only host); other platforms not tested this pass.
- **Regression:** Unknown — the preceding release was used only as the pre-upgrade baseline, not tested for this behaviour on its own.
- **OpenShell issue:** No — the refusal comes from NemoClaw's own rebuild resume path; the gateway is healthy and the sandbox remains registered.

## Environment

```text
Device: x86_64 server, CPU-only (no NVIDIA driver present)
OS: Ubuntu 24.04.4 LTS
Architecture: x86_64
Docker: Docker version 29.5.2
OpenShell CLI: openshell 0.0.116
NemoClaw: v0.0.125 before the upgrade, v0.0.126 after
Agent: OpenClaw
Provider: NVIDIA cloud inference
```

## Steps to Reproduce

1. On a host running the release immediately preceding the one under test, onboard a disposable sandbox and confirm it is Ready. Write a recognizable marker file into its workspace.
2. Start a rebuild and interrupt it once the CLI announces the delete boundary:

```bash
nemoclaw {sandbox} rebuild --yes
```

Interrupt after the line `Deleting old sandbox...` appears and before the rebuild completes. Confirm the command exits nonzero and does not report completion.
3. Confirm a journal was left behind: count the journal or recreate files under the gateway state directory.
4. Upgrade in place to the release under test, and confirm the journal files are still present and unchanged in number.
5. Remove the stale onboard lock the interrupted run left behind, after confirming no onboarding process is running — otherwise every command below is blocked.
6. Run the rebuild again to resume, capturing output and exit code:

```bash
nemoclaw {sandbox} rebuild --yes
```

7. Run `nemoclaw {sandbox} status` and `nemoclaw list`.

## Expected Result

The rebuild resumes the interrupted operation and completes, exits 0, and the sandbox returns to Ready with its workspace marker intact.

If resuming genuinely cannot be supported across an upgrade, the failure should say what the operator can do instead — for example a documented way to discard the stranded journal and start a clean rebuild.

## Actual Result

Step 2, the interrupt, behaves correctly:

```text
boundary announced "Deleting old sandbox..." after 23s
interrupted yes
exit code 130
"rebuild completed" absent
journal files left 7
```

Step 4, the upgrade preserves the journal:

```text
version after upgrade v0.0.126
journal files 7, unchanged
```

Step 6, the resume fails:

```text
Error: The retained rebuild MCP recovery observation is unavailable, so rebuild cannot
safely resume. No sandbox deletion was attempted.

exit code 1
"resum..." in output absent
```

Step 7, nothing was destroyed: the sandbox is still listed and registered exactly once, and no replacement sandbox was created.

## Logs

```text
rebuild (interrupted, pre-upgrade) exit 130, journal written, no completion reported
installer to the release under test CLI updated, journal count unchanged
first resume attempt blocked by the stale onboard lock from step 2
lock removed by hand sandbox visible again in list
second resume attempt exit 1, "retained rebuild MCP recovery observation
is unavailable, so rebuild cannot safely resume.
No sandbox deletion was attempted."
status / list after sandbox present, registered once
```

## Related issues / not duplicate of

No existing report covers a pre-upgrade rebuild journal becoming unresumable after the upgrade.

Related, not duplicates:

- **#10461** — an interrupted onboard leaves a lock whose owner PID is dead. That lock was also hit here and is why step 5 exists, but it is a separate defect: once the lock is removed the resume still fails for a different reason.
- **#11379** — rebuild refuses with "different recreate transaction in progress" when an unrelated sandbox has an incomplete rebuild. That is a cross-sandbox interference report; here only one sandbox exists and the blocker is the missing recovery observation for that same sandbox.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the `nemoclaw {sandbox} rebuild --yes` resume path and the handling of the retained rebuild MCP recovery observation. Then inspect how the gateway state directory and its journal files are preserved during an in-place upgrade. Done means an interrupted rebuild resumes successfully after upgrade, or the failure documents a supported cleanup and fresh-rebuild path without deleting the sandbox unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, typescript
Domain
cli, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.