NVIDIA / NVIDIA/NemoClaw

[DGX Station][Upgrade] upgrade drops the OpenShell gateway registration and never restores it

Open
#11,898 0 comments 0 reactions 1 assignee Claimed by @cjagwani View on GitHub
needs: triage NV QA UAT
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

## Description

An in-place upgrade that crosses an OpenShell gateway generation boundary removes the NemoClaw gateway registration and never puts it back. The installer exits 1, every sandbox is left stopped, and the host cannot be upgraded again: each later installer run now dies earlier than the last one, at the pre-upgrade backup, because there is no gateway to query.

The pre-upgrade backup itself succeeds first, so the user sees their sandboxes captured and then watches the install fail anyway.

The guidance the product prints does not work. It says to ensure OpenShell is running. Starting the gateway's user service does bring the process back up on its port, but the CLI still reports that no gateway is configured, because it is the registration that was deleted, not the process. Recovery took a manual command that appears nowhere in the product output or the documentation.

One sandbox did not survive. After recovery it stayed in an `Error` phase that no supported command could clear: `start` refused it, `rebuild` refused it, and backup failed for every component. It had to be destroyed and recreated, so its state was lost.

The mTLS material and gateway database on disk were intact throughout. Only the CLI-side gateway registration was gone.

- **Platform scope:** Reproduced on DGX Station GB300 (Ubuntu 24.04, aarch64) only; other platforms not tested.
- **Regression:** Unknown — earlier versions not tested for this path.
- **OpenShell issue:** No — the removal happens in the NemoClaw installer's own logged gateway retirement step.

## Environment

```text
Device: NVIDIA DGX Station GB300
OS: Ubuntu 24.04.4 LTS
Architecture: aarch64
Node.js: v22.23.2
npm: 10.9.8
Docker: Docker version 29.6.1, build 8900f1d
OpenShell CLI: 0.0.106 before the upgrade, 0.0.116 after
NemoClaw: v0.0.123 baseline, v0.0.126 target
OpenClaw: 2026.7.1
```

## Steps to Reproduce

Use a host with no previous NemoClaw install: no `nemoclaw`, no `openshell`, no NemoClaw or OpenShell state directories, and the default gateway port free. Do not set any gateway port or gateway state directory override at any point.

1. Install the older release:

```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_INSTALL_TAG=v0.0.123 bash
```

Confirm the OpenShell CLI is 0.0.106.

2. Create three sandboxes: leave two stopped and one running. Confirm all three are Ready and that the gateway is healthy.

3. Run the documented in-place upgrade to a release that ships a newer OpenShell generation:

```bash
curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_INSTALL_TAG=v0.0.126 bash
```

4. When it exits, inspect the gateway:

```bash
systemctl --user is-active nemoclaw-openshell-gateway
openshell status
openshell gateway list
docker ps -a
```

5. Run the identical upgrade command again and compare where it fails.

## Expected Result

The upgrade retires the old gateway, installs the newer OpenShell, registers the gateway again, and completes with exit code 0 and all sandboxes operational.

If the gateway cannot be re-registered, the installer says so and names a command the user can actually run to restore it, and a second attempt is not left in a worse state than the first.

## Actual Result

The backup phase succeeds:

```text
Pre-upgrade backup: 3 backed up, 0 failed, 0 skipped
```

Then the installer removes the gateway and never restores it:

```text
[INFO] Retiring OpenShell 0.0.106 gateway before installing current OpenShell...
[install] openshell 0.0.106 is below minimum 0.0.116 - upgrading...
[install] openshell 0.0.116 installed
[3/3] Onboarding
Failed to query running sandboxes from OpenShell.
OpenShell sandbox inventory error: kind=transport; reason=unreachable; gateway
recovery attempted=yes.
OpenShell could not reach the selected gateway.
NemoClaw tried to recover its OpenShell gateway, but recovery did not complete.
Ensure OpenShell is running: openshell status
[ERROR] Installation incomplete: one or more existing sandboxes failed to upgrade.
exit 1
```

State immediately afterwards:

```text
systemctl --user is-active nemoclaw-openshell-gateway inactive
nothing listening on the gateway port
openshell status "No gateway configured."
openshell gateway list "No gateways found."
all three sandbox containers Exited; the running one
received SIGTERM
```

The systemd journal records a clean shutdown of the gateway, matching the installer's own retirement step. The mTLS material and gateway database under the OpenShell state directory were still present and intact.

Re-running the identical installer command now fails **earlier** and can never recover on its own:

```text
[INFO] Backing up 3 sandbox(es) before upgrading OpenShell...
Failed to query running sandboxes from OpenShell.
OpenShell sandbox inventory error: kind=transport; reason=unreachable; gateway
recovery attempted=yes.
[ERROR] Pre-upgrade backup stopped the installer.
```

The printed remedy does not resolve it. Starting the user service brings the gateway process back up on its port, but the CLI still reports `No gateway configured`, because the registration is what was removed. Recovery required a command the product never mentions:

```bash
openshell gateway add --local --name nemoclaw https://127.0.0.1:8080
```

After recovery, one sandbox was unrecoverable through every supported path:

```text
start "sandbox must be Stopped to start (current phase: Error)"
rebuild "Could not inspect OpenClaw MCP configuration: ... sandbox is not ready"
backup "sandbox-a: backup failed (agents, extensions, workspace, skills, hooks,
canvas, cron, memory, credentials, openclaw.json)"
```

It had to be destroyed and recreated, losing its state.

Why this is not a local environment problem: the host was verified clean before the run (no NemoClaw or OpenShell binaries, no state directories, no containers, gateway port free); the default gateway port was used throughout with no port or state directory override; and the gateway was healthy and serving SSH backups into all three sandboxes seconds before it disappeared.

## Logs

```text
Not captured beyond the terminal output quoted in Actual Result.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.