[Feat] Expose CAO as an ACP server so external hosts can drive a session over a typed protocol

## Overview

Expose CAO as an **ACP server** — an [Agent Client Protocol](https://agentclientprotocol.com) agent that external hosts connect to — so any ACP client (editor, chat host, IDE) can drive a CAO session over a typed protocol instead of the HTTP API plus screen-scraped text.

**This is the opposite direction from the stance already documented in the repo,** and the distinction is the whole proposal:

| Direction | Shape | Status |
|---|---|---|
| **Provider-facing (CAO as ACP client)** | host → CAO → **ACP** → provider CLI | Explicitly not a CAO transport. `docs/grok-cli.md` records that a provider's headless `-p` and ACP modes "are not CAO transports" — CAO drives the interactive TUI through its terminal backend. |
| **Host-facing (CAO as ACP server)** | host → **ACP** → CAO → tmux / herdr → provider CLI | Not filed. What this issue proposes. |

Nothing here asks CAO to change how it drives providers, and nothing here requires a provider to cooperate. CAO keeps its current provider-facing transport unchanged — either terminal backend, tmux or herdr — and presents a structured face *outward*. That is the same shape as the AG-UI work that already shipped (#386, #458): one typed surface over many terminal-mode CLIs.

## Problem

CAO's only programmatic faces today are request/response HTTP, request/response MCP tools, and an AG-UI SSE stream that is deliberately metadata-only. A host that wants CAO to *execute a conversational turn* on its behalf has to:

- poll `GET /terminals/{id}` for a status enum that carries no turn identity,
- read the assistant's reply from `GET /terminals/{id}/output?mode=last`, which is a per-provider regex over the rendered TUI (`extract_last_message_from_script`),
- accept no token streaming — the AG-UI `TEXT_MESSAGE_CONTENT` frame carries `delta=""` by design,
- accept no provider-level tool-call visibility — AG-UI's `TOOL_CALL_START`/`END` describe CAO's own orchestration (`handoff`, `assign`), synthesised from CAO's event log, not the tools the model invoked,
- and cancel by killing the terminal or writing `\x03` into the PTY.

Every host integrating with CAO re-derives that same polling-and-scraping client. Meanwhile the ecosystem has converged on a protocol for exactly this handshake, and **the CLIs CAO already drives are on the ACP registry** — Claude Code, Codex CLI, Gemini CLI, Kiro CLI, Copilot, Goose, OpenCode, Grok Build among ~40 others.

There is a concrete waiting consumer. Kiro Crew is already an ACP *client*: it spawns `kiro-cli acp --agent `, negotiates `initialize` at protocol version `2025-08-22`, and consumes streamed text chunks, `tool_call` / `tool_call_update`, `session/request_permission`, compaction status and `stop_reason` per turn. It has a clean `LLMProvider` abstraction with three registered ACP backends (kiro, claude, kas). Adding a fourth ACP-speaking backend there is a ~200–500 line change following an established pattern — **if** the thing on the other end speaks ACP. Today it cannot, so the only integration available is the ops-MCP tool surface (#581), which manages sessions from the outside but cannot host a turn.

## Why it matters

Without this, "use CAO from an existing agent host" means the host downgrades: a chat UI that streams tokens today would show nothing until a turn completes, then render a regex-extracted block of terminal text. That trade is bad enough that hosts will keep driving provider CLIs directly and CAO's multi-provider orchestration stays invisible to them.

With it, CAO becomes selectable wherever ACP is already spoken, and what the host gains over talking to a single CLI is exactly CAO's differentiator: multi-provider fan-out, `handoff`/`assign` delegation, persistent cross-session memory, workflows, and tool restrictions — presented as one agent.

## User Stories

- As an ACP-capable host, I want to create a CAO session and send a prompt over ACP, so that I do not implement a polling client against CAO's HTTP API.
- As a host, I want streaming assistant output and structured turn-end reasons, so that my UI can render progress rather than waiting for a whole-message read.
- As a host, I want approval prompts delivered as a structured request I answer with a chosen option, so that I do not regex a TUI and inject keystrokes.
- As a host, I want to cancel a turn and get an acknowledged terminal state, so that a stuck turn is recoverable without killing the session.
- As a CAO maintainer, I want capability negotiation to state plainly what CAO can and cannot do, so that a partial implementation is honest rather than silently lossy.

## What maps cleanly, and what does not

This is the load-bearing part of the proposal. ACP has capability negotiation, so a partial implementation is legitimate — but only if the gaps are **declared**, never faked.

| ACP element | CAO today | Assessment |
|---|---|---|
| `initialize` | n/a | New. Advertise a deliberately narrow capability set. |
| `session/new` (`cwd`, `mcpServers`) | `POST /sessions` with working directory, profile, provider | **Maps cleanly.** `mcpServers` passthrough matches how CAO already wires MCP per session. |
| `session/prompt` | `POST /terminals/{id}/input` | **Maps cleanly** as dispatch. |
| `session/load` / `session/resume` / `session/close` | `GET /sessions/{name}`, `DELETE /sessions/{name}` | Mostly maps; replay semantics for `session/load` need a decision (see open questions). |
| `session/request_permission` | `answer_user_prompt`, `ApprovalBridge`, AG-UI interrupt/resume, `WAITING_USER_ANSWER` | **Maps well — the machinery already exists.** Detection is still screen-parsed, which is a fidelity limit, not a blocker. |
| `session/update` → `agent_message_chunk` | No token stream. PTY WebSocket carries raw bytes; `mode=last` is post-hoc | **Approximation only.** Incremental screen-buffer diffs can produce chunks, but they are rendered-output deltas, not model tokens, and they will carry TUI artifacts. Must be labelled as such. |
| `session/update` → `tool_call` / `tool_call_update` | Orchestration-level only (`handoff`/`assign`), synthesised by CAO | **Genuine gap, and not closable from CAO's side.** Provider-level tool calls are invisible across the PTY boundary. Do not advertise this. |
| `session/cancel` + `stopReason: "cancelled"` | Hard-kill terminal, or `\x03` over PTY. `POST /workflows/runs/{id}/cancel` exists for workflows | **Partial.** Best-effort interrupt with an acknowledged stop reason is achievable; a guaranteed clean abort is not. |
| `stopReason` (`end_turn`, `cancelled`, `max_tokens`, `max_turn_requests`, `refusal`) | Status enum only | `end_turn` and `cancelled` are derivable. `refusal` / `max_tokens` are not observable — must not be invented. |
| `usage_update`, `plan` | Not available | Do not advertise. |
| `fs/*`, `terminal/*` client methods | CAO agents use their own tools inside the session | Out of scope; CAO is not delegating file or terminal I/O to the host. |

The honest summary: **session lifecycle and approvals map well, streaming is a degraded approximation, and per-tool observability is impossible without provider cooperation.** An implementation that declares that is useful. One that pretends otherwise is worse than none, because a host will build a UI on events that never arrive.

**Backend nuance — one of these limits is not universal.** CAO has two terminal backends, and they differ on state detection. On **tmux**, status is inferred by pattern-matching pane content. On **herdr**, `get_native_status()` reads herdr's own `agent_status` field via `herdr pane get` and maps its five states (`working`/`blocked`/`done`/`idle`/`unknown`) directly, avoiding pane parsing entirely — so status is native, not scraped. It is not unconditional even there: a wrapped launch command (for example `podman exec`) makes herdr's foreground process the wrapper rather than the agent CLI, so `agent_status` stays `unknown` and CAO falls back to parsing.

What that does **not** change is the two hard limits above. Assistant text still comes from pane content on both backends — `herdr pane read` returns scrollback, and `extract_last_message_from_script` still parses it — so the streaming approximation stands. And neither backend surfaces provider-level tool calls, so the tool-call ceiling stands. The practical consequence for this proposal is that a herdr deployment can report turn state more trustworthily than a tmux one, which is worth knowing when deciding how much the streaming and tool-call gaps actually cost.

## Proposed solution

### Transport shape

ACP's stable transport is stdio with the **client spawning the agent as a subprocess**, not a connection to a long-running daemon. `cao-server` is an HTTP daemon, so the surface needs a thin stdio front-end that translates to CAO's existing API:

```
client
| spawns as subprocess, stdio JSON-RPC 2.0
v
cao-acp-server <-- new
| HTTP
v
cao-server :9889 <-- unchanged
| tmux / herdr
v
provider CLI TUI <-- unchanged
```

This is deliberately the same shape `cao-ops-mcp-server` already has — a stdio protocol server in front of the HTTP API — so it is an established pattern in this repo rather than a new architecture. If ACP's draft streamable-HTTP transport stabilises later, a direct gateway connection becomes possible without changing the layers below.

### Phasing

Phase it so the first slice is small enough to judge.

**Phase 1 — lifecycle only.** A `cao-acp-server` stdio entry point alongside `cao-mcp-server` and `cao-ops-mcp-server`. Implement `initialize`, `session/new`, `session/prompt`, `session/close`, and `session/update` carrying whole-message `agent_message_chunk`s on turn completion. Advertise no streaming, no tool calls. This alone lets an ACP host drive CAO end to end.

**Phase 2 — approvals and cancellation.** `session/request_permission` over the existing approval bridge, and `session/cancel` with a best-effort interrupt returning `stopReason: "cancelled"`.

**Phase 3 — incremental output, if it proves worth it.** Screen-diff chunking behind a capability flag, evaluated against whether the artifacts are tolerable in a real host UI. Reasonable to abandon after Phase 2 if not.

Reuse rather than rebuild: the turn-evidence problem this surface has to solve is the same one `cao-session-liveness` documents (#646) — a status enum cannot identify which turn it belongs to — and `examples/ops-mcp/` (#592) already implements the evidence-based polling that a Phase 1 server would sit on top of.

## Acceptance criteria (Phase 1)

- [ ] `cao-acp-server` runs as a stdio JSON-RPC 2.0 server and completes an `initialize` handshake at ACP protocol version `1`.
- [ ] A host can `session/new` a CAO session with an explicit `cwd` and receive a `sessionId`.
- [ ] `session/prompt` dispatches to the session and the assistant reply returns via `session/update`.
- [ ] The turn ends with an explicit `stopReason`, and the reply corresponds to the prompt that was sent — not a previous turn's output.
- [ ] `session/close` shuts the CAO session down, verified absent afterwards.
- [ ] Capabilities advertised in `initialize` match what is actually implemented; unsupported features are absent rather than stubbed.
- [ ] Protocol-level tests run without provider credentials, in the manner of `test/examples/test_ops_mcp_example.py`.
- [ ] Docs state which ACP features CAO does not implement and why, and `docs/control-planes.md` gains ACP as a fourth inbound surface.

## Alternatives considered

**Keep the ops-MCP surface as the only external control plane (#581, #592).** It manages sessions well — launch, inspect, message, shut down — but a host cannot use it to *host a conversation*: no streaming, no structured approval round-trip, no cancellation. Complementary, not a substitute.

**Extend AG-UI to carry message bodies.** AG-UI's metadata-only posture is deliberate, and its event vocabulary is fleet observability rather than a turn contract. Widening it would grow a CAO-specific protocol where an established one exists, and would not make CAO selectable in hosts that already speak ACP.

**Have hosts talk to provider CLIs over ACP directly, bypassing CAO.** This already works and is why the gap matters: it is the path of least resistance today, and it routes around everything CAO adds.

**Make CAO an ACP client (consume providers over ACP).** A different axis, and the one the repo has already taken a position against in `docs/grok-cli.md`. Worth revisiting separately on its own merits; deliberately out of scope here.

## Non-goals

- Changing how CAO drives providers. Both terminal backends, tmux and herdr, stay as they are.
- Implementing the ACP `fs/*` or `terminal/*` client-side method families.
- Advertising provider-level tool-call events, which the PTY boundary makes unavailable.
- ACP v2 (draft). Target stable version `1`.

## Open questions

1. **`session/load` replay.** ACP expects prior conversation replayed as `session/update` notifications. CAO's history is a rolling screen buffer, not a message list. Replay a best-effort reconstruction, or omit the `loadSession` capability entirely?
2. **Session mapping.** One ACP session per CAO session, or per terminal? A CAO session can hold a conductor plus workers, which has no ACP equivalent — is the supervisor the agent, with delegation invisible to the host?
3. **Multi-provider surfacing.** CAO's differentiator is fan-out across providers. Is that expressed through `session/set_mode`, through profile selection at `session/new`, or left implicit?
4. **Streaming honesty.** Is a screen-diff `agent_message_chunk` acceptable, or does shipping approximate deltas under a protocol whose consumers expect model tokens do more harm than withholding the capability?

## Additional context

**Spec.** [agentclientprotocol.com](https://agentclientprotocol.com) · [github.com/agentclientprotocol/agent-client-protocol](https://github.com/agentclientprotocol/agent-client-protocol) · Apache 2.0 · stable protocol version `1`, v2 in draft · created by Zed Industries, JetBrains co-steward since Oct 2025. Transport is newline-delimited JSON-RPC 2.0 over stdio; streamable HTTP is draft.

**Verification of the current state.** `grep -ri "acp|agent.client.protocol|session/prompt|session/update" src/` returns zero matches — there is no ACP support to extend. The output-extraction boundary was confirmed by reading `providers/base.py`'s `extract_last_message_from_script` contract and its per-provider implementations, and the metadata-only stream by reading the AG-UI event construction where `TEXT_MESSAGE_CONTENT` is emitted with an empty delta. The per-backend state-detection difference was confirmed by reading `backends/base.py`'s `get_native_status()` default (returns `None`, caller falls back to pane parsing) against `backends/herdr_backend.py`'s implementation of it.

**Not verified.** Which optional ACP capabilities each registry agent actually advertises; whether the registry entries are native or adapter-based in every case; and the v2 draft's consolidation of the prompt lifecycle, which may change what a future implementation should target.