NVIDIA / NVIDIA/NemoClaw

[Upgrade] upgrade-sandboxes 0.0.109 -> 0.0.123 (gateway 0.0.101 -> 0.0.106) destroys the sandbox, then fail-closes on a messaging bridge provider collision (legacy "generic" type vs "nemoclaw-mcp-v1"); no supported recovery path

Open
#11,627 0 comments 0 reactions 0 assignees View on GitHub
needs: triage
Dominant language
TypeScript
Stars
22.5k
Forks
3.1k
Avg merge
1d 1h
Merged PRs (30d)
715

Description

### Investigation Summary

- Sandboxes onboarded with v0.0.109/OpenShell 0.0.101 carry messaging bridge
providers registered as Type: generic (verified via `openshell provider get
-telegram-bridge`).
- v0.0.123 requires typed providers: MESSAGING_CREDENTIAL_PROVIDER_TYPE =
"nemoclaw-mcp-v1" (src/lib/messaging/provider-profile.ts:10); the recreate
inspection compares type and fail-closes on mismatch
(src/lib/onboard/gateway-provider-metadata.ts:72,
src/lib/onboard/machine/handlers/sandbox-messaging.ts:300).
- During `nemohermes upgrade-sandboxes`, the rebuild backed up state, DELETED
the old sandbox, then failed at recreate with the collision error — leaving
the sandbox destroyed and only a manual, undocumented recovery possible.
- The documented recovery dead-ends: `nemoclaw channels remove
` requires a Ready sandbox to edit the in-sandbox agent config, but
the old sandbox is destroyed ("sandbox not found") or stuck in Error phase
(old image cannot boot against the upgraded gateway:
HERMES_MCP_CONFIG_DRIFT); `openshell provider delete` is refused while the
provider is attached to the (destroyable) sandbox record.
- Escaped only by `nemoclaw destroy --force` (releases the attachment) +
`openshell provider delete -telegram-bridge` + fresh onboarding with
the bot token re-supplied (provider deletion clears the stored credential).

### Description

Upgrading a host with active messaging channels from v0.0.109/OpenShell 0.0.101
to v0.0.123/OpenShell 0.0.106 via `nemohermes upgrade-sandboxes` cannot
complete: the legacy generic-type bridge providers in the carried-over gateway
state do not match the typed "nemoclaw-mcp-v1" binding the new stack expects,
so every sandbox recreate fail-closes AFTER the old sandbox is destroyed.
Because the only credential-retiring command requires a Ready sandbox, there is
no supported in-CLI path back to a working state without destroying the
sandbox and manually deleting the gateway provider.

Expected: the upgrade path migrates (or authoritatively replaces) legacy
generic-type messaging bridge providers to the typed format, so
upgrade-sandboxes completes for hosts with active channels — or, at minimum,
the recreate's registration path performs an "authorized replacement" of a
type-mismatched legacy provider when the channel credential is re-supplied,
instead of the unconditional collision failure.

Workaround (verified on this host): per sandbox,
1. nemoclaw destroy --force
2. openshell provider delete -telegram-bridge
3. fresh onboard with TELEGRAM_BOT_TOKEN=
(the stored credential is cleared in step 2, so the token must be re-supplied;
workspace state requires a separate snapshot restore if not empty).

### Reproduction Steps

1. With NemoClaw v0.0.109 + OpenShell 0.0.101, onboard a Hermes sandbox with an
active telegram channel (bridge provider registers as Type: generic).
2. Upgrade to v0.0.123 (installer upgrades OpenShell to 0.0.106 and restarts
the gateway, preserving gateway state).
3. Run `nemohermes upgrade-sandboxes`, confirm the rebuild.
4. Observe: state backed up, old sandbox deleted, then
"Sandbox recreate error: Messaging provider '-telegram-bridge' for
sandbox '' does not match the recorded credential binding. No
messaging state was changed." followed by
"Recreate failed after sandbox was destroyed."
5. Attempt the documented recovery
(`nemoclaw channels remove telegram`): fails with
"Failed to read messaging agent config '/sandbox/.hermes/.env': sandbox not
found" (destroyed) or "sandbox '' is not ready (phase: Error)"
(unbootable old image). `openshell provider delete` is refused: "provider is
attached to sandbox(es)".

### Environment

- OS: Fedora Linux 44 (Server Edition), kernel 7.1.13-200.fc44.x86_64, x86_64
- GPU: NVIDIA GeForce RTX 4090 (driver 610.57.04), Docker CDI GPU passthrough
- Docker: Docker Engine 29.7.2 (docker driver)
- NemoClaw / NemoHermes: v0.0.109 -> v0.0.123 (upgraded 2026-09-12)
- OpenShell: 0.0.101 -> 0.0.106 (upgraded by the installer during upgrade-sandboxes)
- Hermes agent runtime: 0.19.0 -> 0.20.6 (sandbox image rebuild)
- Node.js: v24.18.0
- Layout: 2 sandboxes (leryl-01, frankiebox-02) sharing one gateway "nemoclaw" on default port 8080 and one host-local llama.cpp server (llama-cpp-local, ornith:9b,
- 127.0.0.1:8081); each sandbox has a telegram channel.

### Debug Output

[nemoclaw-debug-postfix.tar.gz](https://github.com/user-attachments/files/32147205/nemoclaw-debug-postfix.tar.gz)
[nemoclaw-debug.tar.gz](https://github.com/user-attachments/files/32147204/nemoclaw-debug.tar.gz)

nemoclaw debug --output bundle from this host attached where available
(nemoclaw-debug.tar.gz). Note it was generated after recovery; the failure
sequence below is captured verbatim from the upgrade run.

### Logs

```shell
Legacy provider as registered by 0.0.109 (openshell provider get):

Provider:
Name: frankiebox-02-telegram-bridge
Type: generic
Credential keys: TELEGRAM_BOT_TOKEN

Recreate failure (nemohermes upgrade-sandboxes):

Backing up sandbox state...
✓ State backed up (16 directories, 5 files)
Deleting old sandbox...
✓ Old sandbox deleted
Sandbox recreate error: Messaging provider 'frankiebox-02-telegram-bridge' for sandbox 'frankiebox-02' does not match the recorded credential binding. No messaging state was changed.
Recreate failed after sandbox was destroyed.
Backup is preserved at: /home/mleroy/.nemoclaw/rebuild-backups/frankiebox-02/2026-09-12T05-34-34-730Z

Documented-recovery attempts:

$ nemoclaw frankiebox-02 channels remove telegram
Could not remove 'telegram' from the sandbox agent config: Failed to read messaging agent config '/sandbox/.hermes/.env': Error: × code: 'Some requested entity was not found', message: "sandbox not found"
Channel 'telegram' remains disabled; fix the sandbox config error and re-run

$ nemoclaw leryl-01 channels remove telegram
Could not remove 'telegram' from the sandbox agent config: ... Error: × sandbox 'leryl-01' is not ready (phase: Error); wait for it to reach Ready state

$ openshell provider delete leryl-01-telegram-bridge
Error: × code: 'The system is not in a state required for the operation's execution', message: "provider 'leryl-01-telegram-bridge' is attached to sandbox(es): leryl-01"

Recovery that worked (undocumented):

$ nemoclaw leryl-01 destroy --force
✓ Sandbox 'leryl-01' destroyed
$ openshell provider delete leryl-01-telegram-bridge
✓ (provider gone once the attachment was released by destroy)
$ TELEGRAM_BOT_TOKEN=*** nemoclaw onboard --name leryl-01 # fresh, succeeds

Post-fix, both sandboxes register typed bridges and pass status (v0.0.123).
```

### Checklist

- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate

Contributor guide

Open the contributing guide

Research direction

Start by reproducing `nemohermes upgrade-sandboxes`, then trace the recreate flow through `src/lib/messaging/provider-profile.ts`, `src/lib/onboard/gateway-provider-metadata.ts`, and `src/lib/onboard/machine/handlers/sandbox-messaging.ts`. Verify that legacy generic providers are handled without destroying the sandbox first, and that the upgrade completes with active channels and a recoverable failure path.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.