aws-samples / aws-samples/sample-multi-tenant-ai-agents-on-lambda-microvm

Cross-generation tenant state loss (mount-namespace mismatch) + 2 build-blocking config drifts

Open
#1 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
8
Forks
2
PR merge metrics
No merged PRs in 30d

Description

## Summary

While deploying this sample end-to-end (`src/deploy.sh`, CloudFormation stack, us-east-1, MicroVM image built from `src/microvm/`), I hit three issues. Two are **build-blocking config drift** against current OpenClaw; the third is a **functional bug**: per-tenant EFS-persisted state does **not** survive across MicroVM generations. I traced the third to a **mount-namespace mismatch** between `efs-monitor.sh` and the OpenClaw gateway.

All three were reproduced live; the root cause of the state bug was confirmed with an in-VM probe (details below).

---

## 1. Build fails: `openclaw.json` web-search provider `duckduckgo` not bundled

`src/microvm/openclaw.json`:

```json
"tools": {
"web": {
"search": { "enabled": true, "provider": "duckduckgo" },
"fetch": { "enabled": true }
}
}
```

The `duckduckgo` search provider plugin is not bundled in the current `ghcr.io/openclaw/openclaw:slim` base image, and OpenClaw's strict config validation **aborts the gateway start (exit 1)** when a configured provider has no backing plugin. Because the MicroVM image build runs the gateway during snapshot creation, this fails the whole `create-microvm-image` build.

**Fix that worked:** disable web search (or bundle the plugin):

```json
"search": { "enabled": false }
```

## 2. Build fails: Bedrock plugin install now requires capability consent

`src/microvm/Dockerfile`:

```dockerfile
RUN ... && HOME=/home/node node /app/openclaw.mjs plugins install @openclaw/amazon-bedrock-provider \
...
```

Current OpenClaw added a **capability-consent gate** to `plugins install`; the non-interactive build now exits non-zero waiting for consent.

**Fix that worked:** add `--accept-capabilities`:

```dockerfile
&& HOME=/home/node node /app/openclaw.mjs plugins install @openclaw/amazon-bedrock-provider --accept-capabilities \
```

With fixes #1 and #2, the MicroVM image builds green in ~3.3 min.

---

## 3. Cross-generation tenant state loss — root cause: mount-namespace mismatch

### Symptom
Within a single MicroVM generation, a tenant's conversation state persists correctly. But after the VM is reaped for idleness and a **new generation** is launched for the same tenant, the prior state is gone — the agent behaves as a fresh install. EFS *does* contain the seeded tenant subtree (`/mnt/efs/tenants//...`), so the write side, not the read side, is where state is lost.

### Where it comes from
`src/microvm/efs-monitor.sh` mounts EFS, seeds/adopts the tenant subdir, then binds it over the gateway's state dir:

```bash
mount --bind "$TDIR" "$STATE_DIR" # $STATE_DIR=/home/node/.openclaw
touch "$MARKER"
pkill -f "openclaw.mjs gateway" || true # bounce so gateway restarts against EFS
```

`efs-monitor.sh` and the gateway are **separate background children of `start.sh`**:

```bash
/opt/poc/efs-monitor.sh & # does the mount --bind
...
while true; do node /app/openclaw.mjs gateway ... ; done # the gateway
```

### Root cause (confirmed live)
The `mount --bind` succeeds **in efs-monitor's mount namespace but is invisible to the gateway's namespace.** The two processes do not share a mount namespace (or share only a private, non-propagating one), so a bind established after fork does not propagate to the gateway.

Decisive in-VM probe (run through the agent, since the orchestrator launches VMs with `ALL_INGRESS` and the raw shell needs `SHELL_INGRESS`):

- Running `mount --bind /mnt/efs/tenants/ /home/node/.openclaw` returns **exit 0**, yet
- `mountpoint /home/node/.openclaw` reports **"not a mountpoint" (exit 32)** and
- `mount | grep -c /home/node/.openclaw` returns **0**

So the gateway keeps reading/writing a **namespace-local `/home/node/.openclaw`** that is *not* backed by EFS. Those writes die with the VM generation, while the EFS subtree only ever holds the initial seed → the next generation "adopts" a stale/empty tree.

The `bounce → restart against EFS` design assumes the restarted gateway inherits the bind. It doesn't, because the bind lives in efs-monitor's namespace, not the gateway's.

### Ruled out (red herrings during investigation)
- The adopt-vs-seed marker (`[ ! -f "$TDIR/openclaw.json" ]`) — not the cause; EFS *is* seeded correctly.
- Missing OS capabilities — EFS itself mounts fine (image has `additionalOsCapabilities: [ALL]`).
- Slow NFS `chown` delaying the bind — reordering bind-before-work still shows the bind invisible to the gateway.

### Suggested fixes (not yet validated upstream — sharing as candidates)
1. Establish **shared mount propagation** before the bind, e.g. `mount --make-rshared /` (or `--make-rshared /mnt/efs`) in `efs-monitor.sh` before `mount --bind`, so the bind propagates to peer namespaces.
2. **Bind before the gateway forks its namespace** — do the EFS mount + bind in `start.sh` *before* launching the gateway supervisor, rather than in a concurrent daemon that binds after the gateway is already running.
3. Launch the gateway **inside efs-monitor's namespace** (e.g. same `unshare`/`nsenter` context), so they share the mount view.

Option 1 or 2 is likely the smallest change; both need a rebuild + a cold→regenerate→recall test cycle to confirm.

---

## Environment
- AWS Lambda MicroVMs, us-east-1, arm64, base image `al2023-1`
- Deployed via the repo's own `src/deploy.sh` (CloudFormation)
- Reproduced live; multi-tenant isolation and within-generation state both verified working — only the cross-generation path fails.

Happy to open PRs for #1 and #2 (trivial) and to help validate a fix for #3 if a maintainer has a preferred direction.

Contributor guide

Open the contributing guide

Research direction

Read src/microvm/openclaw.json and src/microvm/Dockerfile first, then trace startup through src/microvm/start.sh and src/microvm/efs-monitor.sh. Reproduce the image build and the cold-to-regenerate-to-recall cycle; done means both configuration failures are resolved and tenant state survives across MicroVM generations.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, shell
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.