[Ubuntu 24.04][Upgrade] status no longer reports the healthy sandbox Ready after an aborted upgrade
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 43m
- Merged PRs (30d)
- 718
Description
## Description
The strict pre-upgrade backup correctly stops an in-place upgrade when one sandbox cannot be captured. After that stop, the sandbox that DID back up cleanly can no longer be reported as Ready. Running `status` on it prints a port-ownership warning, omits the `Phase` line entirely, and ends with "its agent delivery chain could not be proven".
The sandbox itself is fine. At the OpenShell layer it is Ready, its container is up and healthy, SSH into it works, and the registry still lists it with no data loss. Only the user-facing NemoClaw `status` command refuses to say so.
The reason is visible in the product's own output. The installer replaces the NemoClaw CLI with the target version before it runs the backup, by design, and the abort then leaves the older OpenShell gateway in place and still running. The dashboard host port is still held by that retained gateway's forward, and the newly installed CLI cannot prove it started that listener, so it declines to attribute it and degrades the whole status report.
The remedy the message offers is also wrong for this situation. It tells the user to find the port owner, free the port, and rerun `recover`. The port owner is the product's own retained gateway, and the real remedy is to finish the upgrade.
Impact is on trust rather than data. A user who has just been told the upgrade was stopped to protect their sandboxes then checks the sandbox that was protected successfully and is told its delivery chain could not be proven. That reads as a second failure in a flow whose whole purpose is to reassure.
- **Platform scope:** Reproduced on Ubuntu 24.04 x86_64 only; other platforms not tested.
- **Regression:** Unknown — not tested before v0.0.124; seen on both v0.0.124 and v0.0.126.
- **OpenShell issue:** No
## Environment
```text
Device: Ubuntu 24.04 host, x86_64, no GPU
OS: Ubuntu 24.04 LTS
Architecture: x86_64
Node.js: Not on PATH (installer-managed Node.js)
npm: Not on PATH (installer-managed Node.js)
Docker: Docker version 29.8.0
OpenShell CLI: 0.0.106 (docker driver) - the retained gateway; target generation 0.0.116 was never installed because the upgrade aborted first
NemoClaw: v0.0.126 (CLI already replaced before the abort; baseline was v0.0.123)
OpenClaw: 2026.7.1
```
## Steps to Reproduce
1. Install a release whose OpenShell gateway generation differs from the target release's, so the installer will run the strict pre-upgrade backup and would otherwise retire the old gateway:
```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_INSTALL_TAG=v0.0.123 bash
```
Confirm `nemoclaw` reports v0.0.123 and `openshell status` reports Version 0.0.106.
2. Create two managed-image sandboxes on that release:
```bash
nemoclaw onboard --non-interactive --yes --no-gpu --no-sandbox-gpu \
--agent openclaw --name keep-good --fresh
nemoclaw onboard --non-interactive --yes --no-gpu --no-sandbox-gpu \
--agent openclaw --name keep-stuck --fresh
```
Both reach Ready, and `nemoclaw keep-good status` reports `Phase: Ready`.
3. Make `keep-stuck` uncapturable, using a host-level action that is fully reversible:
```bash
docker pause {the keep-stuck container}
```
Confirm SSH into it now fails while `docker ps` still shows it up but paused.
4. Run the pinned in-place upgrade under a TTY:
```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_INSTALL_TAG=v0.0.126 bash
```
The strict backup fires and the installer stops with exit code 1. This part is correct.
5. Run status on the sandbox that backed up cleanly:
```bash
nemoclaw keep-good status
```
## Expected Result
After the upgrade is stopped to protect the user's sandboxes, a sandbox that is healthy and backed up cleanly is still reported as Ready. Its state is unchanged, so `status` should be unchanged too.
If NemoClaw genuinely cannot attribute the dashboard forward in this split state, it says so plainly and names the real remedy — finish the upgrade — rather than telling the user to hunt down and free a port that the product's own retained gateway is holding.
## Actual Result
Step 5, verbatim:
```text
Host port 18789 for 'keep-good' is held by a listener that NemoClaw cannot attribute to
this sandbox's OpenShell forward. NemoClaw cannot prove it started the listener, so it
leaves the listener running and does not restore a forward onto it. Find the owner with
`ss -ltnp 'sport = :18789'` or `lsof -nP -iTCP:18789 -sTCP:LISTEN`, free the port, then
run `nemoclaw keep-good recover` again.
Sandbox-scoped status for 'keep-good':
Sandbox: keep-good
Inference: not verified (gateway/sandbox state not verified)
OpenShell: 0.0.106 (docker)
Harness: OpenClaw (gateway)
Agent: OpenClaw v2026.7.1
Sandbox 'keep-good' is present, but its agent delivery chain could not be proven
(forward-recovery: host port 18789 is held by a listener that NemoClaw cannot attribute
to this sandbox's OpenShell forward, so the dashboard forward was not restored).
Retry `nemoclaw keep-good recover` after addressing the reported layer.
```
There is no `Phase` line and the word `Ready` appears nowhere in the output.
The sandbox is actually healthy at every other layer, checked at the same moment:
```text
openshell sandbox list
keep-good Ready
keep-stuck Error
openshell sandbox get keep-good
Phase: Ready
docker ps -a
the keep-good container is Up and healthy
ssh into the sandbox using its OpenShell ssh-config
succeeds, and the sandbox filesystem is intact
nemoclaw list
still lists both sandboxes, no duplicates, fields unchanged
nemoclaw keep-good snapshot list
the pre-upgrade backup taken during step 4 is present
```
The port owner is the product's own retained gateway:
```text
ss -ltnp 'sport = :18789'
LISTEN 127.0.0.1:18789 users:(("openshell",pid={pid},fd=10))
```
That listener belongs to the retained 0.0.106 gateway user service, whose MainPID and activation timestamp both predate the upgrade attempt and were never changed by it. So the newly installed CLI is declining to attribute a forward that the still-running old gateway legitimately owns.
No post-failure workaround was applied: `recover` was not run, the port was not freed, the upgrade was not retried, and the version state was not reverted.
What worked correctly in the same run, recorded so it is not mistaken for part of this defect: the strict backup counted the uncapturable sandbox, the installer stopped with a clear reason and instruction, the legacy gateway was not retired, and both sandboxes survived with no data loss.
## Logs
```text
Not captured beyond the terminal output quoted in Actual Result.
```
Contributor guide
Research direction
Start by reproducing the aborted upgrade with the pinned installer commands, then compare `nemoclaw keep-good status` with `openshell sandbox get keep-good` and the retained gateway's listener. Trace the status and upgrade-recovery entry points that handle listener attribution. Done means the healthy backed-up sandbox is reported as Ready, and any unresolved split-state message names finishing the upgrade as the remedy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, typescript, ubuntu
- Domain
- cli, devops, release
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100