[Feature]: Reorganise agent-server image build — selectable capabilities, parameterised provider set, and shared eval layers

Problem Statement

The agent-server image bundles every optional capability unconditionally. At 1.64 GB compressed, most of it is capability a given deployment never uses, and the provider set is growing. This issue makes image contents an explicit, selectable contract.

Status: Steps 0–2 done, plus the desktop removal. Step 3 is next and unblocked apart from one product call (does the lean tag ship an editor?).

Step	What	Status
0–1	Provider payload built once, COPY --from into base-image-minimal so the layer is parent-independent	✅ #4651
2	INSTALL_ACP_PROVIDERS and INSTALL_CAPABILITIES build arguments, defaults unchanged	✅ #4687, #4698
3	Publish lean tags; default OSS and Agent Canvas to them	tag published; Canvas PR open — OpenHands/OpenHands#17079, ⛔ blocked on an SDK release
4	Make the lazy npx path solid (cache persistence, --prefer-offline, progress)	✅ #4805
5	Provider additions adopt the mechanism	ongoing
—	Desktop/VNC stack removed entirely (not just made optional)	✅ #4792

Steps 0–2 changed nothing observable: same providers, same pins, same tag contents. Everything is now selectable; nothing published yet exercises the option. Step 3 is the first step where anything moves.

[!NOTE]
First measurement of the published images (amd64), after #4805 merged. Compressed layer totals from the ghcr manifests — what a cold pull transfers:

Tag	Compressed
agent-server:latest-python	1279.2 MB
agent-server:latest-python-slim	954.8 MB
saving	324.4 MB (25.4%)

Consistent with the arm64 A/B figure of 315.3 MB — Codex ships a larger linux-x64 binary. Published slim contents verified: /acp-node empty, all three wrappers absent, node v22.23.2 and npx present for the fallback, Chromium/Docker/OpenVSCode intact.

Still reaching no user. Agent Canvas adoption is open as OpenHands/OpenHands#17079 (reviewed: "well-scoped and correct, no substantive correctness or architecture issues"), but it cannot merge — see The one thing blocking Step 3 below.

The one thing blocking Step 3

Everything measured in this issue reaches a user only when Agent Canvas points at the slim tag, and that PR (OpenHands/OpenHands#17079) cannot merge until an SDK release publishes <version>-python-slim.

config/defaults.json pins versions.agentServer to a release tag. 1.44.1-python-slim is a 404 — the slim variant merged after 1.44.1 (2026-08-28), which is still the newest release. latest-python-slim and main-python-slim exist today, but a commit-SHA pin is rejected by scripts/check-sdk-version-sync.mjs, which cross-validates versions.agentServer against the SDK version openhands-automation was built on. So the release is a genuine dependency, not a formality.

The release is overdue on its own merits: main is 14 commits ahead of v1.44.1, including fix(sdk): remove secrets from subprocess env (#4801), fix(tools): mask secrets in every tool's observation at the shared chokepoint (#4788) and #4805 itself. Releases here are cut by manually dispatching prepare-release.yml with a version; nobody has. Once <version>-python-slim publishes, #17079 is a one-line bump and CI goes green.

A second gate is independent of the release and needs a person, not a commit: the reviewer asked for a human maintainer to sign off on moving ACP provider acquisition from preinstalled to first-use npx, since that changes a deployment-affecting default. Worth starting in parallel so the two gates clear together.

What the image is made of

Every figure below is a controlled A/B build — build with the component, build without it, diff the compressed layer totals from an --output type=oci export. linux/arm64, --target base-image, reproduced on two separate commits. Compressed is what a cold pull actually transfers; on-disk is what it occupies.

Component	Compressed	On disk	Gated by
base-image-minimal — the floor	343.8 MB	—	not optional
GitHub CLI	13.7 MB	37.1 MB	ungated — no off switch
OpenVSCode Server	60.8 MB	—	INSTALL_CAPABILITIES=vscode
Docker Engine	135.5 MB	354.2 MB	INSTALL_CAPABILITIES=docker
Chromium	270.3 MB	~562 MB tree	INSTALL_CAPABILITIES=browser
ACP providers (all three)	315.3 MB	977 MB	INSTALL_ACP_PROVIDERS
Total (default)	1133.6 MB

Build totals: floor 343.8 · +caps, no ACP (lean) 818.3 · +caps −docker 682.8 · default 1133.6 MB.

So roughly 70% of the image is optional payload, and the floor is 343.8 MB.

Inside the ACP payload (977 MB on disk → 315.3 MB compressed):

Part	On disk	Measured alone*
Node 22 runtime (shared)	192 MB	—
codex-acp	320 MB	186.5 MB
claude-agent-acp	312 MB	148.2 MB
gemini-cli	157 MB	91.9 MB

* each includes the shared Node 22 runtime, so they do not sum to 315.3 MB — installing all three amortises it once (~56 MB of the difference). Note claude-agent-acp is a 0.5 MB npm package that pulls 312 MB of Claude Code + agent SDK, and codex-acp is 1.1 MB pulling a ~320 MB platform binary.

[!WARNING]
Two earlier figures in this issue were wrong, both derived from published-image layer history rather than controlled builds, and both over-counted. 564.9 → 400.9 MB for editor+browser+desktop; 601.6 → 315.3 MB for ACP. Manifest attribution is not reliable here — measure by A/B build. All figures above are A/B; the amd64 equivalents have not been measured and should not be extrapolated from these.

[!NOTE]
Chromium is 270.3 MB, not the 169.4 MB previously recorded — and the number changed because of #4792. The old figure was measured while the desktop stack was still installed: Chromium and XFCE shared GTK3/X11, mesa, and icon-theme dependencies, so Chromium's marginal cost looked small. With desktop gone, Chromium alone carries them — chromium 286 MB + chromium-common 82 MB + libllvm19 118 MB + mesa-libgallium 33 MB + libgtk-3-common 30 MB + adwaita-icon-theme 13 MB on disk. Removing the desktop moved cost onto the browser rather than eliminating all of it. The 72.0 MB saving #4792 measured is still real.

Desired Behavior

Step 3 — publish lean tags and move OSS + Agent Canvas. Enterprise stays put; evaluation stays preloaded. Canvas Docker E2E green is the gate.

[!NOTE]
The lean tag is now fully specified. Details in OHE-3179.

• Desktop/VNC — removed entirely in #4792.
• Chromium — kept, but the margin is thinner than first thought: at 270.3 MB (not 169.4) it is nearly as large as the whole ACP payload. Kept because a Chromium-less image silently removes the agent's browser tool — a capability loss, not just a size trade — and BrowserToolSet degrades by disappearing rather than explaining itself.
• OpenVSCode Server — kept. (decided) It is the only way a user can inspect or edit the agent's workspace from Canvas — stock openvscode-server on port 8001, opened at ?tkn=…&folder=workspace. At 61.1 MB it is ~5% of the image; cutting a user-visible capability for that muddies an otherwise purely additive change.

So: lean = INSTALL_CAPABILITIES=vscode,browser,docker with INSTALL_ACP_PROVIDERS= empty. Measured on base-image, arm64: 1133.6 MB → 818.3 MB, a 315.3 MB / 28% saving for the median user, who never selects an ACP agent.

Docker Engine stays in both the default and slim images — decided, see Docker in Canvas below. It was a candidate at 135.5 MB, but the agent being able to run containers is worth more than the size.

Still to confirm: benchmarks and evaluation are unaffected (they build from -minimal targets, which bypass the capability stage).

Step 4 — a lean user who picks an ACP agent gets told why, and gets it downloaded. (decided)

The behaviour: when a selected provider is not present in the running image, surface a message saying it is missing because this is the lean image, then start the download automatically and show progress. If it fails, point at the ACP-preloaded tag. Not a silent stall, and not a dead end.

The fallback mechanism already exists and is version-pinned — resolve_acp_command() leaves npx -y <pkg>@<ver> in place when the binary is absent — and it is viable in the lean image: verified node v22.23.2 and npx present from the base image, /acp-node empty, all three wrappers absent.

Measured, not estimated. Cold npm install of each provider in a clean Node 22 container:

Provider	Cold install	On disk
gemini-cli	1 s	94 MB
codex-acp	3 s	321 MB
claude-agent-acp	4 s	314 MB

[!NOTE]
This corrects an earlier claim in this issue. It previously said a cold Codex fetch was "very unlikely" to fit the 90 s acp_startup_timeout, extrapolated from unpacked size. Measured, it is 3 seconds on broadband. Note also that all three are heavy on disk — claude-agent-acp is a 0.5 MB npm package that pulls ~314 MB, so the top-level package size is not a useful proxy.

So acp_startup_timeout (90 s, a hard deadline covering subprocess spawn) is a slow-connection problem, not a universal one: each provider is roughly 100 MB over the wire, so below about 10 Mbps Codex approaches or exceeds it and fails as "ACP startup timed out" — which reads like a hang rather than a download.

Work required, in value order:

• Persist the npm cache to a durable, conversation-independent path. Today it is container-local, so docker run --rm re-downloads every time. Highest value, and mostly configuration. Note acp_isolate_data_dir may relocate HOME per conversation, so set npm_config_cache explicitly rather than relying on HOME.
• --prefer-offline in the registry default commands so a warm cache needs no network. Compatible with _parse_npx_invocation, which already skips leading npx flags.
• Keep the install off the hard deadline — warm the cache in an explicit step before the handshake, with its own budget and a log line, so slow links degrade into a visible wait rather than a timeout.
• Explain the not-preinstalled case and, on failure, name the ACP-preloaded tag.

Scope note: the npx fallback only works where the base image ships Node. The python base does (v22.23.2); the java and golang bases may not, so a lean build of those would have no npx at all — ACP unavailable rather than slow. ACP remains a python-variant concern only.

No new capability protocol is needed. One already exists end to end: BrowserToolSet.is_usable() → list_usable_tools() → ServerInfo.usable_tools → Canvas's isAgentServerToolAvailable() → shouldIncludeTool(). ServerInfo also has an unused capabilities: list[str] field if per-provider advertisement is ever wanted.

Step 5 — provider additions use the mechanism. #4714 (Kimi Code) already does this correctly: registry-only, no Dockerfile change. #4419 (Pi) is the outlier, still pre-installing unconditionally, and should join INSTALL_ACP_PROVIDERS. Same applies to #4624, #4634, #4635, #4639.

Docker in Canvas — decided: it stays

Decision: the agent running inside Docker and running Docker itself — spinning up containers, building a Dockerfile and running it, bringing up a docker compose stack — is worth more than the 135.5 MB. #4808 (shim explaining absence) and #4809 (drop Docker from slim) are both closed.

What was established while deciding:

• It works today under --privileged — verified against the published latest-python, daemon 29.7.2 starts normally. In the documented Canvas launch it cannot start (error creating default "bridge" network: operation not permitted), so the capability was real but undiscoverable.
• No lighter runtime avoids the privilege requirement. Rootless dockerd with seccomp=unconfined, apparmor=unconfined, --device /dev/fuse and --device /dev/net/tun starts the daemon fully ("Daemon has completed initialization", server 29.7.2), but containers it creates fail on error mounting "proc" to rootfs: operation not permitted — identically under BuildKit and the legacy builder, and with both overlayfs and vfs. runc needs CAP_SYS_ADMIN to mount /proc, granted by the host at container start and unobtainable later. Same wall for rootless BuildKit. (Tested on Docker Desktop for macOS; a native Linux host with unprivileged user namespaces may differ.)
• Enterprise depends on it and is unaffected either way: saas-deploy sets RUNTIME_CLASS: "sysbox-runc" in production, staging and development; Sysbox is installed via ArgoCD in every environment; Replicated exposes "Sandbox Isolation" as a customer-facing option defaulting to sysbox, with an install guide at enterprise/k8s-install/sysbox.mdx.

Follow-ups:

• Documentation — the capability was written up only under enterprise/, so a self-hoster had no way to find it. Fixed in OpenHands/docs#774: documents --privileged for Canvas with the security tradeoff, and cross-links both ways with the Enterprise guide.
• A trim, considered and not taken. Dropping compose + buildx saves 36.1 MB (31%) and docker build / docker run still work — verified, the daemon serves the legacy builder when buildx is absent. Not pursued: BuildKit is materially better and compose is one of the documented use cases. docker-ce-rootless-extras (23.5 MB on disk, pulled in as a Recommends) was also skipped — surgical removal costs more Dockerfile complexity than it saves, and a blanket --no-install-recommends would also drop git, procps, openssh-client and systemd.

Constraints

[!WARNING]
Three things that are easy to break and expensive to discover late.

Evaluation and benchmarks never lazy-install. They pass a bare acp_command, which _prefer_pinned_binary does not rewrite, so there is no fallback — and runtime downloads mean per-pod network, non-reproducible runs, and broken air-gap. They stay preloaded at every step. Measure pull time and pod-start latency, not bytes.

Enterprise does not move. It works today: pulled once per node, cached by the kubelet, no network at conversation start. There is no measured pain justifying a migration, and the node-level layer cache is already a good cache. What protects enterprise from the image doubling as providers are added is the pinned default list from Step 2 — not moving it anywhere.

Registry membership is not image membership. Registering a provider in ACP_PROVIDERS makes it selectable; it must not put it in the image. A provider enters the default list only by a deliberate edit to that line.

No per-conversation image selection. It is the intuitive design and it is not available. Canvas's Dockerfile is FROM ${AGENT_SERVER_IMAGE} AS final — "All-in-one: Agent Server + Automation + Frontend" — so the agent-server is the Canvas container and its capability set is fixed when the user pulls. On cloud, get_agent_server_image() is @cache-memoized from process env (one image per deployment), and runtime-api's warm pools deliberately strip per-conversation env so one pool serves any conversation — provider-specific pods would mean maintaining two pools. Choose images by consumer and tag, never per conversation.

Also load-bearing: the provider payload must stay at top-level /acp-node. A destination the parent image already owns (anything under /opt) breaks layer dedup.

Remaining opportunities, after ACP

Ranked by value over effort. Everything here is additive to the steps above.

1. Delete benchmarks/utils/Dockerfile.agent-layer-commit0; have commit0 consume eval-base. ✅ Done — OpenHands/benchmarks#778 merged 2026-09-02; the file is gone from main and commit0 now builds through the same three-phase pipeline as SWE-Bench. That removes the last place a benchmark pinned ACP versions independently of the SDK registry.

2. Plumb agent_type into benchmark image builds so default runs — the large majority — stop baking ACP they never touch.

3. Gate the GitHub CLI (13.7 MB). The only component with no off switch. Trivial, low value.

4. The uv layer dedupes per upstream repo, not globally — ~169 MB per benchmark set. Low value; listed so it is not rediscovered and overestimated again.

[!WARNING]
This issue previously claimed ~7 GB per 300-instance set, which was wrong by roughly 40×. That estimate came from two eval images whose uv layers differed — without checking that they came from different upstream repos. Measured across 764 published SWE-bench instances: the layer is 21.17 MB (not 24.5), and there are 8 distinct upstream repos, so the real waste is about 169 MB, not gigabytes. Same-repo instances share the layer (django-12155 and django-11999 are both e48d9c06…); different repos do not (scikit-learn e5f46351…, sympy 2febb744…). It is keyed by parent base image, which is per-repo. For contrast the /agent-server layer is bb3acf5c… across all of them — genuinely global, which is what #4651 achieved.

The fix would be the #4651 technique (copy to a fresh top-level path instead of /bin, which the parent already owns). This got cheaper: one of the two cross-repo consumers is gone, since benchmarks#778 deleted Dockerfile.agent-layer-commit0 and with it its COPY --from=builder /bin/uv /bin/uvx /bin/. Only apptainer_build.py's /usr/local/bin/uv remains. Worth re-costing rather than assuming the old verdict.

[!NOTE]
chromium-headless-shell was investigated and rejected — measured saving is 4.9 MB, not "materially smaller". Same base image, arm64, compressed layer totals:

Build	Total	Marginal
bare base	199.8 MB	—
+ apt chromium (today)	470.6 MB	+270.8 MB
+ playwright chromium-headless-shell	465.7 MB	+265.9 MB

Two reasons the intuition failed. The shell binary is not self-contained — it fails on launch with libglib-2.0.so.0: cannot open shared object file and 20 missing shared libraries, so it needs playwright install --with-deps, which installs the same system stack apt chromium pulls (libllvm19 117.6 MB, mesa-libgallium 33.4 MB, GTK/X11). And --with-deps adds ~85 MB of fonts apt chromium does not (fonts-unifont 31.8, fonts-wqy-zenhei 16.0, fonts-freefont-ttf 14.1, fonts-ipafont-gothic 11.9, fonts-noto-color-emoji 10.8), which cancels most of the smaller binary.

This is the same lesson as the 169.4 → 270.3 MB correction: the browser's cost is dominated by shared system libraries, not the browser binary, so swapping the binary does not touch the expensive part. Treat ~270 MB as the practical floor for a working browser on this base image. The ~10-line _playwright_chromium_paths discovery fix is not worth 4.9 MB plus a runtime download and a changed font set.

Not low-hanging despite the size: Docker Engine (see the callout above — it works under --privileged, so removal needs the shim and a migration window) and a slimmer base image (the 343.8 MB floor is largely nikolaik/python-nodejs including a gcc/g++ toolchain that native npm builds may need).

Acceptance Criteria

• ACP providers and the editor/browser/Docker capabilities are independently selectable at build time (#4687, #4698)
• Desktop/VNC removed entirely rather than left as a selectable dead capability (#4792)
• Provider set is one list argument, defaulting to today's three, with no published tag changing contents (#4687)
• The provider payload is shared across eval instances rather than rebuilt per instance, verified by layer digest (#4651)
• The consumer questions are answered from the code — desktop removed, keep Chromium, OpenVSCode is a product call (OHE-3179)
• The OpenVSCode call is made — kept in the lean tag (61.1 MB; only in-product workspace editor)
• Cold-npx timing measured against the 90 s acp_startup_timeout — 1–4 s on broadband, so the deadline is a slow-connection concern only
• The npm cache survives container replacement, verified by a second run that does not re-download (#4805 — second run 0.08 s)
• Selecting a provider absent from the running image explains why (lean image), starts the download with visible progress, and offers the preloaded tag on failure
• Published images measured on amd64 — see the note under Problem Statement. Per-component amd64 A/B still outstanding; component figures are arm64 only and must not be extrapolated
• Docker decision made — stays in both images; #4808/#4809 closed, documented in OpenHands/docs#774. Was: ship a shim, then decide whether the lean tag excludes Docker Engine — 135.5 MB, unusable in the documented launch but working under --privileged, and a marketed enterprise feature
• A lean Python image is published and Agent Canvas builds against it with Docker E2E green
• Existing tags keep their contents for at least one migration release before any default flips
• Per-component amd64 A/B, so the retained components can be priced on the arch that ships. (The old "≤ 0.65 GB default image" target was dropped from this issue along the way. It was written before Docker Engine, Chromium and OpenVSCode were each deliberately kept — those three are the bulk of the remaining 954.8 MB, so the target was traded away by explicit decisions rather than missed. Quantifying that split on amd64 is the open work; the component figures on record are arm64 and must not be extrapolated.)
• agent_type reaches benchmark image builds so default runs produce images without ACP
• Dockerfile.agent-layer-commit0 is deleted and commit0 consumes eval-base; no benchmark pins ACP versions independently of the SDK registry (OpenHands/benchmarks#778, merged 2026-09-02)
• Evaluation pull time and pod-start latency do not regress
• npm cache survives container replacement via a documented volume; a warm cache starts a provider without network
• A provider that cannot be found produces an actionable message, not an opaque exec failure (#4805)
• Enterprise runtime pods are unchanged and keep node-level layer caching

Related

• ✅ #4651, #4687, #4698 — merged; Steps 0–2.
• ✅ #4792 — merged; removed the desktop/VNC stack entirely (+81/−1174).
• 🔄 #4805 — open; ACP version parity test + runtime version logging, being extended to cover the slim variant and the npx fallback (Step 4).
• OHE-3179 — the Step 3 delivery ticket (publish the lean tag, default Canvas and OSS to it).
• docs/enterprise/docker-in-sandbox.mdx — why Docker Engine cannot simply be removed the way the desktop stack was.
• ✅ #4645 — closed by #4698; its two consumer questions carried into Step 3 above.
• #4644 — closed, superseded by OpenHands/architecture#19 (private), which owns vendor terms and the provider admission policy.
• #4624, #4419, #4714, #4634 / #4635 / #4639 — provider additions that should adopt the mechanism.