ACP downstream CI: what each consuming repo must assert against the SDK registry

Problem

The SDK is now the single source of ACP provider truth — ACP_PROVIDERS plus ACP_INSTALL_CATALOG (#4832) — and #4820 is about to add four more providers. But every consuming repo hand-maintains its own copy of provider knowledge, and none of them fail when the registry changes. A provider lands in the SDK and is silently absent downstream.

The bug history is almost entirely downstream, and every one was found by a human reading a failed production run rather than by CI:

benchmarks#634	acp_agent_name/acp_agent_version empty for a slice of gemini rows despite ACP running
benchmarks#736	codex-acp silently ignoring OPENAI_BASE_URL → -32603 across a run; first misdiagnosed as agent_type not propagating, costing a full investigation
benchmarks#608 / #635 / #609	proxy_cost=0 twice, and cost recorded against acp-managed instead of the real model
benchmarks#488	workspace_keepalive not preventing idle-kill failures
benchmarks#625	running the deprecated @zed-industries/claude-agent-acp

Filing this as one issue rather than a per-repo trio (cf. #4308 coordinating #4304 / OpenHands#16161 / enterprise#96) so the effort stays in one place.

Desired Behavior

One contract, four consumers.

The SDK publishes the registry — ACP_PROVIDERS for Python consumers, the @openhands/typescript-client mirror for TS. Full-coverage consumers assert equality against it. Product clients that offer a subset declare SURFACED_ACP_PROVIDERS independently and assert only the subset direction.

For Canvas, SURFACED_ACP_PROVIDERS is derived from the opt-in ACP_PROVIDER_UI map. A surfaced key that the client registry renames or removes breaks loudly, while a registry addition is a no-op. Leak tests derive the unsurfaced set from the pinned registry; a complementary not-surfaced list would turn this into a partition and couple every upstream addition back into Canvas.

Out of scope: intra-SDK hardening (#4830), and the loose/floating-dependency audits (#4304 / OpenHands#16161 / enterprise#96), which are a different failure class.

Acceptance Criteria

OpenHands/benchmarks — imports openhands-sdk directly, so these are plain pytest.

• _ACP_COMMANDS, _ACP_ENV_VARS (benchmarks/utils/acp.py) and the agent_type choices (benchmarks/utils/args_parser.py:109) all cover ACP_PROVIDERS keys as acp-<key>
• _ACP_COMMANDS["acp-<key>"] == [binary_name, *trailing_args_from_default_command] — ["gemini", "--acp"] is hardcoded today, so a binary_name change makes every eval instance fail opaquely
• _ACP_ENV_VARS["acp-<key>"] == [api_key_env_var, base_url_env_var]
• setup_acp_workspace()'s provider-specific branch (claude-only today) is registry-driven or on an explicit list
• Post-run assertions on the harness's own output, one per closed bug: every output.jsonl row has non-empty acp_agent_name/acp_agent_version (#634); acp_agent_version equals the pinned version, proving the image shipped the pin rather than an npx fallback; cost > 0 (#608/#635/#609); zero "Stopping idle runtime" (#488)

OpenHands/evaluation — YAML + shell; CI already runs a pytest + pyyaml job.

• agent_type choices identical across eval-job.yml, run-eval-harness.yml and run-infer.yml, and eval-job.yml's free-form type: string description is not a fourth hand-kept copy (it currently omits acp-gemini)
• agent_type propagation asserted end to end: eval-job.yml → run-infer.yml (--set env.AGENT_TYPE) → kube_job.yaml → run_*.sh. #736 spent an entire investigation establishing this by hand
• The hardcoded provider env exports in eval-job/scripts/run_infer.sh:33-38 (ANTHROPIC/OPENAI/GEMINI pairs) derive from — or are asserted against — api_key_env_var/base_url_env_var, so a new provider's credentials are wired automatically. The comment above them is already stale, naming only claude and codex

OpenHands/OpenHands (canvas) — consumes the ts-client mirror.

• SURFACED_ACP_PROVIDERS, derived from ACP_PROVIDER_UI (src/constants/acp-providers.ts), is asserted ⊆ set(ACP_PROVIDERS) — a rename or removal breaks loudly, an addition is a no-op
• Auth probes and onboarding secret fields are gated by SURFACED_ACP_PROVIDERS; leak tests derive unsurfaced providers from the pinned client registry rather than maintaining a complementary list
• Every surfaced provider's description_key resolves in i18n
• tests/e2e/live-acp/ runs in CI — it is already written and credentialed, just never wired to a workflow
• config/defaults.json constraints.agentClientProtocol derives from the SDK's own bound once #4830 P0-5 lands, instead of canvas independently carrying the SDK's risk

OpenHands/enterprise

Cloud ships a deliberate subset. Requirement: registering a harness upstream must not change Cloud or Replicated behaviour at all — Cloud should not have to know a harness exists in order not to offer it.

[!NOTE]
Amended 2026-09-04. Previously a supported / not-supported split asserted to partition ACP_PROVIDERS. A partition forces this repo to enumerate every harness the SDK registers — the coupling #4841 removes. Now a subset assertion against one declared set. Implementation: OpenHands/enterprise#315.

• CLOUD_ACP_PROVIDERS declared and asserted ⊆ set(ACP_PROVIDERS) — a rename or removal breaks loudly, an addition is a no-op
• The web-client config injector emits exactly that set; it iterates ACP_PROVIDERS.values() unfiltered today (default_web_client_config_injector.py:320-335)
• A provider outside the set fails closed at conversation start — covering persisted, org-settings and direct-API values, since ACPServerKind widens with the registry
• For each provider in the set: the acpserver tag projection resolves it, acp-provider-secrets.ts covers its api_key_env_var and every file_secrets[*].secret_name, the org/member overlay handles its kind, and it has a credential preflight (only codex has one today)
• The typed alternative (collapsing agent_kind / the tag / AgentType into one column) was considered and rejected in OpenHands/enterprise#306

Absorbed: wiring the merged providers downstream

The criteria above make a downstream gap fail. These make the newly-merged providers actually usable there. Absorbed from #4635 (Pi) and #4639 (OpenCode), which each carried an identical eval-wiring list; keeping one copy here rather than one per provider.

Kimi (#4714, merged) and Pi (#4419, merged) are in ACP_PROVIDERS today and wired nowhere downstream. OpenCode joins them when #4827 merges — and note that PR auto-closes #4639 while deliberately excluding this wiring, which is why it is recorded here rather than left there to disappear.

Per provider, in benchmarks / evaluation:

• Agent type, command, provider environment and workflow option added to the existing benchmark/evaluation path
• The declared model and a per-attempt LiteLLM key configured headlessly, rejecting unexpected primary, auxiliary or fallback models
• Stored configuration, authentication, sessions, plugins and caches prevented from leaking between runs — via the provider's data_dir_env_var, which for both gemini-cli and OpenCode is plain HOME (see #4639), so relocating a scoped variable is not sufficient
• The reported agent identity and version recorded, and acp_agent_version asserted equal to the pinned version — proving the image shipped the pin rather than silently falling back to npx
• A five-instance shared smoke covering read, edit, test, reconciled telemetry, and a second clean run

Per-provider gotchas — established when each landed, worth having before wiring rather than discovering in a run:

Provider	Watch out for
Kimi	available_models=() and default_model=None — the CLI picks its own model, so there is no list to validate against. api_key_env_var=None, so check how it takes credentials before assuming the usual API-key + base-URL pair routes through the LiteLLM proxy. Catalog pins 0.38.0; npm latest is 0.41.0.
Pi	Two independent pins — pi-acp (adapter) plus @earendil-works/pi-coding-agent (engine). @earendil-works/pi-coding-agent is pinned at 0.83.0 against 0.85.0 on npm.
OpenCode	OPENCODE_CONFIG_DIR points at a config file only; its state follows XDG_{DATA,CONFIG,CACHE}_HOME, each falling back to HOME — so HOME is the only single lever that relocates all of it, as with gemini-cli. Per-platform binary, ~185 MB on linux.

None of the three is in benchmarks/utils/acp.py today — it still knows only acp-claude, acp-codex and acp-gemini, and args_parser.py offers the same three.

[!NOTE]
These were P0 for #4627's paired comparison. That epic has been dormant since 2026-08-26; its measurement-specific children (#4638 telemetry, #4637 scheduling, #4642 report) stay there and are not absorbed here — they are a measurement product, not downstream enablement. Hermes (#4634) is not absorbed either, but for a different reason: it is not merged yet. It will be a normal registry provider (installed via uvx from a pinned git tag rather than npm), so once it lands the registry-keyed criteria above apply to it exactly as they do to the others.

Cross-cutting

• The SDK ships one importable coverage helper so the four repos assert the same contract instead of reimplementing it
• SDK version-bump PRs into each repo run that repo's ACP coverage tests, so a bump that adds a provider fails there rather than in production

Related: #4830 (intra-SDK hardening) · #4841 (where the Cloud supported-set is declared) · OpenHands/enterprise#306 (typed agent identity — rejected, see there) · #4832 / #4828 (install catalog) · #4820 (four incoming providers) · #4823 · OpenHands/automation#414

This issue update was made by an AI agent (OpenHands) on behalf of the issue author.

Start with benchmarks/utils/acp.py, benchmarks/utils/args_parser.py, and the evaluation workflow files named in the criteria, then compare their provider handling with ACP_PROVIDERS. Run the existing pytest and pyyaml CI job and inspect tests/e2e/live-acp/; done means all four consumers assert their declared coverage, downstream wiring is tested, and version bumps fail when registry coverage breaks.