aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

bug(agent image mismatch): arm64 agent-image build breaks on x86 hosts after reboot — binfmt registration is ephemeral (structural fix: native ARM builder)

Open
#571 1 comment 0 reactions 0 assignees View on GitHub
agent-runtime bug enhancement infra-cdk P1
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

### Component

Infra / build & deploy (agent image build; `mise //cdk:deploy`) + onboarding docs

### Describe the feature

Make the arm64 agent-image build survive a fresh/rebooted host — either by not depending on QEMU emulation at all (build natively on ARM), or by documenting a *persistent* binfmt registration. Today the only guidance (#431) is a one-shot manual `binfmt --install` that silently disappears on the next reboot or Docker-daemon restart.

### Use case

The agent image targets `linux/arm64` (Graviton). On an **x86_64** host, `mise //cdk:deploy` needs QEMU/binfmt emulation or the build dies with `exec /bin/sh: exec format error`.

#431 (closed) added a docs pre-empt step:

```
docker run --privileged --rm tonistiigi/binfmt --install arm64
```

But that command writes handlers into the kernel's `/proc/sys/fs/binfmt_misc/` — **runtime kernel state, not on-disk config**. It is wiped by:

- a reboot (a cloud dev box stopping/starting, or a fresh instance), and
- a Docker daemon restart.

So a deployer who followed #431 and got a green deploy comes back after a reboot to the *same* `exec format error`, with no doc hint that the registration is ephemeral. Live-hit on a cloud dev box: binfmt registered in an earlier session, gone after the box restarted, deploy failed again with the truncated `exec format error`.

This blocks any non-Graviton deployer intermittently and is the last remaining piece of the "make a fresh deploy succeed without gap-by-gap rework" arc (bootstrap least-privilege #403/#405/#408/#410 + docs #431).

### Proposed solution

Preferred (structural — removes the emulation dependency entirely):
- Build the agent image on a **native ARM64 builder** (Graviton runner / CodeBuild ARM fleet) so x86 hosts never touch binfmt. Wire it into `deploy.yml` / the deploy task so both CI and local x86 deploys get a native build.

Fallback (if emulation must stay for local x86):
- Document a **persistent** binfmt registration instead of the one-shot:
- Debian/Ubuntu: `sudo apt install -y qemu-user-static binfmt-support` (installs the `systemd-binfmt` service that re-applies `/etc/binfmt.d/` every boot).
- Amazon Linux: `sudo yum install -y qemu-user-static`, or a `@reboot`/systemd unit that re-runs the `tonistiigi/binfmt --install arm64` command.
- In `QUICK_START` + the `/setup` skill, add a one-liner: "this registration does **not** survive a reboot or Docker restart — use the persistent install, or a Graviton host, so you don't re-hit `exec format error`."

### Other information

- Related: #431 (docs pre-empt, closed — covered the manual step but not durability or the structural build) and the bootstrap least-privilege cluster #403/#405/#408/#410.
- Not a stack bug — the CDK/image is correct; this is a host-environment/build-topology gap.
- Surfaced while bringing the context-gated ECS compute substrate live (`--context compute_type=ecs`), but the `linux/arm64` image build (and therefore this blocker) is independent of the ECS vs AgentCore substrate choice.

### Acknowledgements

- [ ] I may be able to implement this feature

Contributor guide

Open the contributing guide

Research direction

Start with the `mise //cdk:deploy` entry point and inspect `deploy.yml`, `QUICK_START`, and the `/setup` skill to trace how the arm64 image is built on x86 hosts. Determine whether the build wiring supports a native ARM builder or needs persistent binfmt guidance. Done means a fresh or rebooted host can complete the arm64 deploy without recurring `exec format error` failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, typescript
Domain
build-system, devops, documentation, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.