aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
bug(agent image mismatch): arm64 agent-image build breaks on x86 hosts after reboot — binfmt registration is ephemeral (structural fix: native ARM builder)
- Dominant language
- TypeScript
- Stars
- 143
- Forks
- 46
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 20
Description
### Component
Infra / build & deploy (agent image build; `mise //cdk:deploy`) + onboarding docs
### Describe the feature
Make the arm64 agent-image build survive a fresh/rebooted host — either by not depending on QEMU emulation at all (build natively on ARM), or by documenting a *persistent* binfmt registration. Today the only guidance (#431) is a one-shot manual `binfmt --install` that silently disappears on the next reboot or Docker-daemon restart.
### Use case
The agent image targets `linux/arm64` (Graviton). On an **x86_64** host, `mise //cdk:deploy` needs QEMU/binfmt emulation or the build dies with `exec /bin/sh: exec format error`.
#431 (closed) added a docs pre-empt step:
```
docker run --privileged --rm tonistiigi/binfmt --install arm64
```
But that command writes handlers into the kernel's `/proc/sys/fs/binfmt_misc/` — **runtime kernel state, not on-disk config**. It is wiped by:
- a reboot (a cloud dev box stopping/starting, or a fresh instance), and
- a Docker daemon restart.
So a deployer who followed #431 and got a green deploy comes back after a reboot to the *same* `exec format error`, with no doc hint that the registration is ephemeral. Live-hit on a cloud dev box: binfmt registered in an earlier session, gone after the box restarted, deploy failed again with the truncated `exec format error`.
This blocks any non-Graviton deployer intermittently and is the last remaining piece of the "make a fresh deploy succeed without gap-by-gap rework" arc (bootstrap least-privilege #403/#405/#408/#410 + docs #431).
### Proposed solution
Preferred (structural — removes the emulation dependency entirely):
- Build the agent image on a **native ARM64 builder** (Graviton runner / CodeBuild ARM fleet) so x86 hosts never touch binfmt. Wire it into `deploy.yml` / the deploy task so both CI and local x86 deploys get a native build.
Fallback (if emulation must stay for local x86):
- Document a **persistent** binfmt registration instead of the one-shot:
- Debian/Ubuntu: `sudo apt install -y qemu-user-static binfmt-support` (installs the `systemd-binfmt` service that re-applies `/etc/binfmt.d/` every boot).
- Amazon Linux: `sudo yum install -y qemu-user-static`, or a `@reboot`/systemd unit that re-runs the `tonistiigi/binfmt --install arm64` command.
- In `QUICK_START` + the `/setup` skill, add a one-liner: "this registration does **not** survive a reboot or Docker restart — use the persistent install, or a Graviton host, so you don't re-hit `exec format error`."
### Other information
- Related: #431 (docs pre-empt, closed — covered the manual step but not durability or the structural build) and the bootstrap least-privilege cluster #403/#405/#408/#410.
- Not a stack bug — the CDK/image is correct; this is a host-environment/build-topology gap.
- Surfaced while bringing the context-gated ECS compute substrate live (`--context compute_type=ecs`), but the `linux/arm64` image build (and therefore this blocker) is independent of the ECS vs AgentCore substrate choice.
### Acknowledgements
- [ ] I may be able to implement this feature
Contributor guide
Research direction
Start with the `mise //cdk:deploy` entry point and inspect `deploy.yml`, `QUICK_START`, and the `/setup` skill to trace how the arm64 image is built on x86 hosts. Determine whether the build wiring supports a native ARM builder or needs persistent binfmt guidance. Done means a fresh or rebooted host can complete the arm64 deploy without recurring `exec format error` failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, docker, typescript
- Domain
- build-system, devops, documentation, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100