shields down / channels add --force can leave a sandbox in a permanently stuck mutation containment lock
- Dominant language
- TypeScript
- Stars
- 22.5k
- Forks
- 3.1k
- Avg merge
- 1d 1h
- Merged PRs (30d)
- 715
Description
## Description
After a normal, documented shields-down/mutation sequence (either "shields down" followed by an exec call while still inside the timeout window, or a "channels add --force" retry on a sandbox), the sandbox enters a permanently stuck "Sandbox mutation containment" state. Every subsequent per-sandbox command fails with the same containment error, and destroy hangs indefinitely instead of either succeeding or reporting the same error.
**Platform scope:** Reproduced on Ubuntu 26.04 only; other platforms not tested.
**Regression:** Unknown — earlier versions not tested for this specific trigger.
**OpenShell issue:** No — the containment-lock mechanism and its lifecycle are NemoClaw's own code.
## Environment
```text
Device: Ubuntu 26.04 x86_64 server, NVIDIA GPU (GPU not relevant to this failure)
OS: Ubuntu 26.04 LTS
Architecture: x86_64
Node.js: v22.23.2
npm: 10.9.8
Docker: 29.7.2
OpenShell CLI: 0.0.106
NemoClaw: v0.0.116
OpenClaw: N/A - Hermes Agent v0.19.0
```
## Steps to Reproduce
1. Onboard a NemoHermes sandbox with Slack configured (any provider/model).
2. Run: `nemoclaw {sandbox} shields down --reason "..." --timeout 5m` (succeeds, exit 0).
3. While still inside the 5-minute shields-down window, run any sandbox-mutating command, e.g. `nemoclaw {sandbox} exec -- hermes send --to slack:{channel} "test"`.
4. Observe the containment error on this and all subsequent `{sandbox}`-scoped commands (`channels status`, `shields status`, `connect --probe-only`, `doctor`, `exec`).
Alternate trigger observed: on a fresh sandbox, run `nemoclaw {name} channels add slack --force` immediately after an aborted (conflict-detected) `channels add slack` attempt on the same sandbox — the retry itself produces the same containment error before completing.
## Expected Result
Sandbox mutation locking should be transparent to the user and self-recovering, or at minimum recoverable via a documented command such as `doctor --fix` or `destroy`.
## Actual Result
The lock never clears on its own — confirmed by waiting for the shields-down timer to fire and exit naturally, after which the containment error still persisted. `doctor` cannot run at all (blocked by the same containment error). `destroy --yes` hangs indefinitely with zero output (confirmed over 90+ seconds via both captured stdout and a live interactive session) instead of either succeeding or reporting the same containment error.
This defect cascaded into blocking further testing entirely, since the only available credential for a related check became stuck to an unrecoverable sandbox.
## Logs
```text
Error: Sandbox mutation containment is active for '{sandbox}' at
'{state-dir}/mcp-lifecycle-locks/{hash}.lock.containment' (generation token
'{redacted}'). A previous owner or stale-lock reaper exited without proof that
every descendant stopped. Stop all NemoClaw processes for this sandbox; inspect
'{hash}.lock', '{hash}.lock.reaper', and '{hash}.lock.deadline'; record each
target's file kind, device/inode, and owner token when present; verify those
identities and this containment token are unchanged; remove only those exact
stale owner generations first and this exact containment generation last
before retrying.
destroy --yes: no output at all after 90+ seconds; process still running, no exit.
```
## Related Bugs
This is one of several distinct, previously-reported triggers into the same general sandbox-mutation-containment mechanism, each fixed or tracked independently rather than as one root cause: GitHub issue #8877 (a headless optional-channel prompt exiting inside the mutation lock escalates to durable containment), GitHub issue #9750 (an auto-restore timer race leaves a stale containment lock after shields-down; fixed and verified in a recent release), GitHub issue #10066 (a stranded registry entry with no backing container recreates the containment lock destroy refuses on), and GitHub issue #10094 (a shields-down sandbox enters a permanent "process-tree containment" lock after an auto-restore timeout). This report's trigger — a mutating command issued during an active shields-down window, or a forced channel-add retry — does not match any of those specific documented trigger conditions, so it is filed as a new, distinct entry point into the same recurring class of defect rather than a duplicate of any one of them.
Contributor guide
Research direction
Start by tracing the sandbox mutation-containment lifecycle from the `shields down`, `channels add --force`, and sandbox-scoped command entry points, using the reported lock paths and containment error as guides. Reproduce both triggers on Ubuntu and inspect how `doctor` and `destroy --yes` handle the resulting state. Done means the lock does not remain permanently after either trigger and recovery commands no longer hang or become unusable.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cli, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100