[Upgrade] upgrade-sandboxes 0.0.109 -> 0.0.123 (gateway 0.0.101 -> 0.0.106) destroys the sandbox, then fail-closes on a messaging bridge provider collision (legacy "generic" type vs "nemoclaw-mcp-v1"); no supported recovery path
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
### Investigation Summary
- Sandboxes onboarded with v0.0.109/OpenShell 0.0.101 carry messaging bridge
providers registered as Type: generic (verified via `openshell provider get
-telegram-bridge`).
- v0.0.123 requires typed providers: MESSAGING_CREDENTIAL_PROVIDER_TYPE =
"nemoclaw-mcp-v1" (src/lib/messaging/provider-profile.ts:10); the recreate
inspection compares type and fail-closes on mismatch
(src/lib/onboard/gateway-provider-metadata.ts:72,
src/lib/onboard/machine/handlers/sandbox-messaging.ts:300).
- During `nemohermes upgrade-sandboxes`, the rebuild backed up state, DELETED
the old sandbox, then failed at recreate with the collision error — leaving
the sandbox destroyed and only a manual, undocumented recovery possible.
- The documented recovery dead-ends: `nemoclaw channels remove
` requires a Ready sandbox to edit the in-sandbox agent config, but
the old sandbox is destroyed ("sandbox not found") or stuck in Error phase
(old image cannot boot against the upgraded gateway:
HERMES_MCP_CONFIG_DRIFT); `openshell provider delete` is refused while the
provider is attached to the (destroyable) sandbox record.
- Escaped only by `nemoclaw destroy --force` (releases the attachment) +
`openshell provider delete -telegram-bridge` + fresh onboarding with
the bot token re-supplied (provider deletion clears the stored credential).
### Description
Upgrading a host with active messaging channels from v0.0.109/OpenShell 0.0.101
to v0.0.123/OpenShell 0.0.106 via `nemohermes upgrade-sandboxes` cannot
complete: the legacy generic-type bridge providers in the carried-over gateway
state do not match the typed "nemoclaw-mcp-v1" binding the new stack expects,
so every sandbox recreate fail-closes AFTER the old sandbox is destroyed.
Because the only credential-retiring command requires a Ready sandbox, there is
no supported in-CLI path back to a working state without destroying the
sandbox and manually deleting the gateway provider.
Expected: the upgrade path migrates (or authoritatively replaces) legacy
generic-type messaging bridge providers to the typed format, so
upgrade-sandboxes completes for hosts with active channels — or, at minimum,
the recreate's registration path performs an "authorized replacement" of a
type-mismatched legacy provider when the channel credential is re-supplied,
instead of the unconditional collision failure.
Workaround (verified on this host): per sandbox,
1. nemoclaw destroy --force
2. openshell provider delete -telegram-bridge
3. fresh onboard with TELEGRAM_BOT_TOKEN=
(the stored credential is cleared in step 2, so the token must be re-supplied;
workspace state requires a separate snapshot restore if not empty).
### Reproduction Steps
1. With NemoClaw v0.0.109 + OpenShell 0.0.101, onboard a Hermes sandbox with an
active telegram channel (bridge provider registers as Type: generic).
2. Upgrade to v0.0.123 (installer upgrades OpenShell to 0.0.106 and restarts
the gateway, preserving gateway state).
3. Run `nemohermes upgrade-sandboxes`, confirm the rebuild.
4. Observe: state backed up, old sandbox deleted, then
"Sandbox recreate error: Messaging provider '-telegram-bridge' for
sandbox '' does not match the recorded credential binding. No
messaging state was changed." followed by
"Recreate failed after sandbox was destroyed."
5. Attempt the documented recovery
(`nemoclaw channels remove telegram`): fails with
"Failed to read messaging agent config '/sandbox/.hermes/.env': sandbox not
found" (destroyed) or "sandbox '' is not ready (phase: Error)"
(unbootable old image). `openshell provider delete` is refused: "provider is
attached to sandbox(es)".
### Environment
- OS: Fedora Linux 44 (Server Edition), kernel 7.1.13-200.fc44.x86_64, x86_64
- GPU: NVIDIA GeForce RTX 4090 (driver 610.57.04), Docker CDI GPU passthrough
- Docker: Docker Engine 29.7.2 (docker driver)
- NemoClaw / NemoHermes: v0.0.109 -> v0.0.123 (upgraded 2026-09-12)
- OpenShell: 0.0.101 -> 0.0.106 (upgraded by the installer during upgrade-sandboxes)
- Hermes agent runtime: 0.19.0 -> 0.20.6 (sandbox image rebuild)
- Node.js: v24.18.0
- Layout: 2 sandboxes (leryl-01, frankiebox-02) sharing one gateway "nemoclaw" on default port 8080 and one host-local llama.cpp server (llama-cpp-local, ornith:9b,
- 127.0.0.1:8081); each sandbox has a telegram channel.
### Debug Output
[nemoclaw-debug-postfix.tar.gz](https://github.com/user-attachments/files/32147205/nemoclaw-debug-postfix.tar.gz)
[nemoclaw-debug.tar.gz](https://github.com/user-attachments/files/32147204/nemoclaw-debug.tar.gz)
nemoclaw debug --output bundle from this host attached where available
(nemoclaw-debug.tar.gz). Note it was generated after recovery; the failure
sequence below is captured verbatim from the upgrade run.
### Logs
```shell
Legacy provider as registered by 0.0.109 (openshell provider get):
Provider:
Name: frankiebox-02-telegram-bridge
Type: generic
Credential keys: TELEGRAM_BOT_TOKEN
Recreate failure (nemohermes upgrade-sandboxes):
Backing up sandbox state...
✓ State backed up (16 directories, 5 files)
Deleting old sandbox...
✓ Old sandbox deleted
Sandbox recreate error: Messaging provider 'frankiebox-02-telegram-bridge' for sandbox 'frankiebox-02' does not match the recorded credential binding. No messaging state was changed.
Recreate failed after sandbox was destroyed.
Backup is preserved at: /home/mleroy/.nemoclaw/rebuild-backups/frankiebox-02/2026-09-12T05-34-34-730Z
Documented-recovery attempts:
$ nemoclaw frankiebox-02 channels remove telegram
Could not remove 'telegram' from the sandbox agent config: Failed to read messaging agent config '/sandbox/.hermes/.env': Error: × code: 'Some requested entity was not found', message: "sandbox not found"
Channel 'telegram' remains disabled; fix the sandbox config error and re-run
$ nemoclaw leryl-01 channels remove telegram
Could not remove 'telegram' from the sandbox agent config: ... Error: × sandbox 'leryl-01' is not ready (phase: Error); wait for it to reach Ready state
$ openshell provider delete leryl-01-telegram-bridge
Error: × code: 'The system is not in a state required for the operation's execution', message: "provider 'leryl-01-telegram-bridge' is attached to sandbox(es): leryl-01"
Recovery that worked (undocumented):
$ nemoclaw leryl-01 destroy --force
✓ Sandbox 'leryl-01' destroyed
$ openshell provider delete leryl-01-telegram-bridge
✓ (provider gone once the attachment was released by destroy)
$ TELEGRAM_BOT_TOKEN=*** nemoclaw onboard --name leryl-01 # fresh, succeeds
Post-fix, both sandboxes register typed bridges and pass status (v0.0.123).
```
### Checklist
- [x] I confirmed this bug is reproducible
- [x] I searched existing issues and this is not a duplicate
Contributor guide
Research direction
Start by reproducing `nemohermes upgrade-sandboxes`, then trace the recreate flow through `src/lib/messaging/provider-profile.ts`, `src/lib/onboard/gateway-provider-metadata.ts`, and `src/lib/onboard/machine/handlers/sandbox-messaging.ts`. Verify that legacy generic providers are handled without destroying the sandbox first, and that the upgrade completes with active channels and a recoverable failure path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100