aws-samples / aws-samples/sample-multi-tenant-ai-agents-on-lambda-microvm
Cross-generation tenant state loss (mount-namespace mismatch) + 2 build-blocking config drifts
- Dominant language
- Python
- Stars
- 8
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
While deploying this sample end-to-end (`src/deploy.sh`, CloudFormation stack, us-east-1, MicroVM image built from `src/microvm/`), I hit three issues. Two are **build-blocking config drift** against current OpenClaw; the third is a **functional bug**: per-tenant EFS-persisted state does **not** survive across MicroVM generations. I traced the third to a **mount-namespace mismatch** between `efs-monitor.sh` and the OpenClaw gateway.
All three were reproduced live; the root cause of the state bug was confirmed with an in-VM probe (details below).
---
## 1. Build fails: `openclaw.json` web-search provider `duckduckgo` not bundled
`src/microvm/openclaw.json`:
```json
"tools": {
"web": {
"search": { "enabled": true, "provider": "duckduckgo" },
"fetch": { "enabled": true }
}
}
```
The `duckduckgo` search provider plugin is not bundled in the current `ghcr.io/openclaw/openclaw:slim` base image, and OpenClaw's strict config validation **aborts the gateway start (exit 1)** when a configured provider has no backing plugin. Because the MicroVM image build runs the gateway during snapshot creation, this fails the whole `create-microvm-image` build.
**Fix that worked:** disable web search (or bundle the plugin):
```json
"search": { "enabled": false }
```
## 2. Build fails: Bedrock plugin install now requires capability consent
`src/microvm/Dockerfile`:
```dockerfile
RUN ... && HOME=/home/node node /app/openclaw.mjs plugins install @openclaw/amazon-bedrock-provider \
...
```
Current OpenClaw added a **capability-consent gate** to `plugins install`; the non-interactive build now exits non-zero waiting for consent.
**Fix that worked:** add `--accept-capabilities`:
```dockerfile
&& HOME=/home/node node /app/openclaw.mjs plugins install @openclaw/amazon-bedrock-provider --accept-capabilities \
```
With fixes #1 and #2, the MicroVM image builds green in ~3.3 min.
---
## 3. Cross-generation tenant state loss — root cause: mount-namespace mismatch
### Symptom
Within a single MicroVM generation, a tenant's conversation state persists correctly. But after the VM is reaped for idleness and a **new generation** is launched for the same tenant, the prior state is gone — the agent behaves as a fresh install. EFS *does* contain the seeded tenant subtree (`/mnt/efs/tenants//...`), so the write side, not the read side, is where state is lost.
### Where it comes from
`src/microvm/efs-monitor.sh` mounts EFS, seeds/adopts the tenant subdir, then binds it over the gateway's state dir:
```bash
mount --bind "$TDIR" "$STATE_DIR" # $STATE_DIR=/home/node/.openclaw
touch "$MARKER"
pkill -f "openclaw.mjs gateway" || true # bounce so gateway restarts against EFS
```
`efs-monitor.sh` and the gateway are **separate background children of `start.sh`**:
```bash
/opt/poc/efs-monitor.sh & # does the mount --bind
...
while true; do node /app/openclaw.mjs gateway ... ; done # the gateway
```
### Root cause (confirmed live)
The `mount --bind` succeeds **in efs-monitor's mount namespace but is invisible to the gateway's namespace.** The two processes do not share a mount namespace (or share only a private, non-propagating one), so a bind established after fork does not propagate to the gateway.
Decisive in-VM probe (run through the agent, since the orchestrator launches VMs with `ALL_INGRESS` and the raw shell needs `SHELL_INGRESS`):
- Running `mount --bind /mnt/efs/tenants/ /home/node/.openclaw` returns **exit 0**, yet
- `mountpoint /home/node/.openclaw` reports **"not a mountpoint" (exit 32)** and
- `mount | grep -c /home/node/.openclaw` returns **0**
So the gateway keeps reading/writing a **namespace-local `/home/node/.openclaw`** that is *not* backed by EFS. Those writes die with the VM generation, while the EFS subtree only ever holds the initial seed → the next generation "adopts" a stale/empty tree.
The `bounce → restart against EFS` design assumes the restarted gateway inherits the bind. It doesn't, because the bind lives in efs-monitor's namespace, not the gateway's.
### Ruled out (red herrings during investigation)
- The adopt-vs-seed marker (`[ ! -f "$TDIR/openclaw.json" ]`) — not the cause; EFS *is* seeded correctly.
- Missing OS capabilities — EFS itself mounts fine (image has `additionalOsCapabilities: [ALL]`).
- Slow NFS `chown` delaying the bind — reordering bind-before-work still shows the bind invisible to the gateway.
### Suggested fixes (not yet validated upstream — sharing as candidates)
1. Establish **shared mount propagation** before the bind, e.g. `mount --make-rshared /` (or `--make-rshared /mnt/efs`) in `efs-monitor.sh` before `mount --bind`, so the bind propagates to peer namespaces.
2. **Bind before the gateway forks its namespace** — do the EFS mount + bind in `start.sh` *before* launching the gateway supervisor, rather than in a concurrent daemon that binds after the gateway is already running.
3. Launch the gateway **inside efs-monitor's namespace** (e.g. same `unshare`/`nsenter` context), so they share the mount view.
Option 1 or 2 is likely the smallest change; both need a rebuild + a cold→regenerate→recall test cycle to confirm.
---
## Environment
- AWS Lambda MicroVMs, us-east-1, arm64, base image `al2023-1`
- Deployed via the repo's own `src/deploy.sh` (CloudFormation)
- Reproduced live; multi-tenant isolation and within-generation state both verified working — only the cross-generation path fails.
Happy to open PRs for #1 and #2 (trivial) and to help validate a fix for #3 if a maintainer has a preferred direction.
Contributor guide
Research direction
Read src/microvm/openclaw.json and src/microvm/Dockerfile first, then trace startup through src/microvm/start.sh and src/microvm/efs-monitor.sh. Reproduce the image build and the cold-to-regenerate-to-recall cycle; done means both configuration failures are resolved and tenant state survives across MicroVM generations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, shell
- Domain
- cloud, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100