[Feat] Carry terminal output, input and status across the pod boundary created by #745

[Feat] Carry terminal output, input and status across the pod boundary created by #745
Part of #777 (CAO 3.0). Created by #745 — that issue centralizes cao-server, and this one carries what then has to cross the gap.

## Why this exists

**3.0's bar is: keep today's capability, but host it remotely in a native way.** This issue carries terminal viewing, input and status across the boundary introduced by #745. It does not introduce a message broker. Today's event bus already drops events when a subscriber queue fills (`services/event_bus.py:172-195`), so preserving its interface is not proof that output or completion reaches its destination.

## What actually breaks

CAO has two distinct terminal paths that currently require local execution access. The background output/status path is:

```
tmux session → pipe-pane → FIFO (named pipe) → fifo_reader → event_bus → status_monitor → consumers
```

- `fifo_reader.py:173` calls `os.mkfifo()`. `FIFO_DIR` is `CAO_HOME_DIR/fifos` (`constants.py:130`) — a local filesystem path.
- `event_bus.py:1` is, by its own docstring, an **in-process** pub/sub bus.
- `status_monitor.py` is a *consumer* of `terminal.{id}.output` and derives status from it, so it inherits the same constraint — **and it is worse than that.** Status detection does not work from the output stream alone: `_process_chunk` runs provider status detection which, for tmux-backed providers, *"shells out to the `tmux` binary via libtmux"* (`services/status_monitor.py:170-178`), and the stale-pane checks call `capture-pane` directly (`:736`, `:892`, `:930`). So the status monitor needs a **local tmux socket**, not just the bytes.

**The browser terminal is a separate path, not an event-bus subscriber.** `/terminals/{id}/ws` asks the backend for an attach command, starts it in a local PTY, forwards raw bytes to the browser, and writes browser input and resize events back to that PTY (`api/main.py:6901-7003`). `TerminalView.tsx` sends both input and resize messages (`web/src/components/TerminalView.tsx:46-104`). Forwarding FIFO output alone does not preserve this interactive session, its initial screen, or its resize behaviour.

Today this is satisfied because **each worker pod runs its own `cao-server`** (`elastic/entrypoint.sh`), so tmux, the FIFO, the reader and the bus are all co-located in one pod. That co-location is why the current design works.

#745 centralizes cao-server into one server workload. The tmux session stays in the worker pod; without the execution bridge, the central server cannot open that worker's FIFO or use its tmux socket. The existing local consumers therefore cannot receive or derive remote output/status unchanged.

Three capabilities are lost:

| Capability | Today | After #745 |
|---|---|---|
| Watch and resize a browser terminal | works through a local backend attach process and PTY | the server cannot attach to the remote terminal |
| Send input to an agent | API input uses backend operations; browser input writes to the attached PTY | both execution paths are in another pod |
| Observe terminal status | works (derived from output, plus direct `capture-pane` probes) | source is gone |

## Why a shared filesystem does not solve it

The EKS example already mounts EFS, so this is a reasonable thing to reach for. It does not work here, for two independent reasons:

1. **Named pipes do not function across NFS/EFS clients.** A FIFO's data is passed inside the kernel and the filesystem entry "merely serves as a reference point" ([`fifo(7)`](https://man7.org/linux/man-pages/man7/fifo.7.html)). NFS/EFS stores only the special-file inode, so a writer on host A and a reader on host B use different in-kernel pipe objects and never exchange data. Unix domain sockets are likewise same-machine-only ([`unix(7)`](https://man7.org/linux/man-pages/man7/unix.7.html)). EFS documents which special-file types can be *created*, but documents no cross-client IPC for them.
2. **Tmux control still requires its local socket.** `pipe-pane` arranges redirected output, while operations such as `capture-pane`, `send-keys` and interactive attach still communicate with the tmux server. Sharing a storage path does not relay those operations. Counts of command-name strings, including diagnostic text and comments, are not evidence of distinct execution paths.

The state volumes are not the shared EFS workspace. The supervisor uses a `ReadWriteOnce` gp3 claim (`examples/cao-clusters/kubernetes/eks/supervisor.yaml:179-187`); each worker uses its own `emptyDir` (`examples/cao-clusters/kubernetes/eks/broker.py:446-449`). Both keep FIFO handling local. EFS is the separate workspace volume.

## What to build

**Reuse the in-process bus and existing topic contracts where possible.** Local FIFO capture and tmux operations stay beside the agent. Add a persistent **outbound** connection from each worker to cao-server's Service address, carrying the background stream and the interactive attach/control path with explicit routing:

```
worker pod cao-server pod
tmux → FIFO → fifo_reader ───────► republish → in-process bus → consumers
◄─────── input (send-keys), control
```

- output and status stream **up** to cao-server, which republishes them onto its own bus;
- input, terminal resize and control travel **down** the connection and execute beside the agent;
- the existing browser WebSocket remains client-facing, but its backend attach path is relayed to the correct worker rather than launching a local attach process;
- consumers that use only the bus contract can be reused; code that reads provider state or calls the execution backend must be adapted;
- the tmux-calling code stays co-located with tmux, which is the constraint that actually matters.

**Local use is unchanged.** No new dependency, no second code path — remote is the same pipeline with a network hop where the co-located hop used to be.

**The native CLI is a separate client too.** Interactive `cao launch` currently invokes `get_backend().attach_session(...)` on the client machine (`cli/commands/launch.py:310-337`). Relay that supported interaction to the authorized remote runtime rather than requiring the client's tmux to own the session. Keep terminal sizing, input, output and detach behavior consistent with the browser, through the same execution-side implementation. #745 owns the broader client/operation matrix, including the existing manual snapshot-restore contract.

### Which side derives status

This issue cannot claim that existing consumers carry on untouched. They cannot: the status monitor shells out to tmux, and after #745 the tmux socket lives only in the worker. Forwarding raw output to an unchanged server-side monitor would leave it calling a tmux that isn't there.

**In remote mode, status is derived in the worker, next to the tmux socket, and sent up as an explicit status message.** cao-server maintains the current status from worker reports and reconnect snapshots, and republishes status events for its consumers. It must not keep calling the local provider detector when completion polling asks for a remote terminal's status. Local mode retains its existing detector.

That means this issue owns a decision it had previously left implicit — for each operation, which side runs it:

| Operation | Runs where |
| --- | --- |
| Provider construction and config files | Worker |
| Launch and readiness | Worker |
| Raw output capture (`pipe-pane`) | Worker |
| **Status derivation and stale-pane probes** | **Worker** — needs the tmux socket |
| Input and key injection (`send-keys`) | Worker |
| Interactive browser attach, byte input and terminal resize | Worker; cao-server relays the existing browser WebSocket |
| Last-message extraction (`capture-pane`) | Worker |
| Graceful exit and cancellation | Worker |
| Republishing onto the event bus | cao-server |
| Status lookup/completion polling | cao-server, using the correct worker's current status rather than local tmux probes |
| Consumers with no local execution dependency | cao-server, retaining their existing contracts where possible |

The rule in remote mode is simple: **anything that touches the tmux socket runs in the worker.** The shared server relays interactive traffic and maintains the state its existing consumers need. Work that assumes the server can reach tmux has to move; keeping a topic name does not make that work disappear.

### Gaps must be visible

#745 requires *"ordered output/event delivery, bounded buffering, and explicit gaps or partial output."* Give each output stream a **generation and monotonic position**, bound to the correct terminal and execution. Background FIFO capture and an interactive PTY attachment are different streams; their positions are not interchangeable. A new worker/stream cannot reset a counter and have its bytes mistaken for a continuation of the old one.

It buys two things the design otherwise cannot deliver:

- a viewer that reconnects resumes from its last position instead of silently skipping whatever arrived while it was away;
- a real gap is **reported** rather than absorbed, so "the agent went quiet" is distinguishable from "I lost the text where it said what went wrong."

The position must describe captured output before a lossy queue can discard it, and remain meaningful through worker forwarding, server fan-out and browser resume. A counter added only after loss cannot reveal the missing bytes. Define an end position or heartbeat watermark so loss of the final chunk is detectable even when no later output arrives.

Bounded replay is for the output stream, not an unlimited delivery guarantee. A reconnect inside the retained window can replay; an expired window or lost generation must produce an explicit gap/reset. The browser currently has neither automatic reconnect nor a resume cursor, so the client changes belong here too.

**Output replay is separate from execution control.** #745 still requires correlated commands and acknowledgements, retained completion results before cleanup, and a distinction between cancellation requested and execution stopped. Carry those states explicitly; do not route them solely through the best-effort output bus or blindly resend a launch/input command after losing its response. This needs no general-purpose message broker and makes no exactly-once promise for external effects.

Terminal replay also does not come from an MCP session or SSE feature by assumption. Follow #745's chosen component/protocol compatibility contract; the terminal's own generation/position and operation identity survive independently of transport request identifiers.

Runtime/assignment generation fencing is distinct from prompt-turn freshness within a still-running agent. The latter is the inherited issue #735: relaying the latest status/response does not prove it belongs to the latest prompt, and this transport work must not be used alone to close that issue.

### Relevant transport precedents

These systems provide precedents for direct interactive channels; their direction and infrastructure differ:

| System | Transport | Direction |
|---|---|---|
| [Coder](https://github.com/coder/coder/blob/main/agent/agent.go#L1181-L1189) | WebSocket/dRPC tailnet | agent dials out to coderd |
| [AWS SSM Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) | WebSocket control + data channel | agent dials out to relay |
| [Amazon ECS Exec](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-exec.html) | same as SSM | dials out from task |
| [GitHub Codespaces](https://docs.github.com/en/codespaces/reference/security-in-github-codespaces) | SSH over TLS WebSocket | codespace dials relay |
| [kubectl exec/attach](https://github.com/kubernetes/enhancements/tree/master/keps/sig-api-machinery/4006-transition-spdy-to-websockets) | WebSocket (SPDY being retired) | control plane dials in |

The design choice for CAO is to keep its existing in-process coordination and use direct runtime channels, without a mandatory message broker. That choice does not depend on proving that no other production system has ever carried terminal data through a broker.

### Why not use the Kubernetes exec API instead

`pods/exec` and `pods/attach` are alternatives, but would put product terminal traffic through the Kubernetes control plane and require execution permissions beyond the broker's existing boundary. #745 chooses outbound runtime channels instead, keeping ordinary agent operation independent of cluster exec access. Do not justify this with a universal four-hour session limit; timeout, proxy and capacity behavior must be established for the actual cluster. Pod log following also does not supply the bidirectional attach/input/resize contract.

## Constraint: cao-server runs as a single replica

This design assumes **one** cao-server replica, and that is a real precondition rather than an oversight. With two or more replicas behind one Service, a worker's uplink and a browser's `/terminals/{id}/ws` viewer can land on **different replicas**, and the viewer's replica has no path to that terminal. A per-replica in-process bus provably breaks.

[Coder](https://github.com/coder/coder/blob/main/tailnet/coordinator.go#L113-L135) — the closest architectural twin — says exactly this in its own source: the in-memory coordinator *"is incompatible with multiple Coder replicas as all node data is in-memory."* Its HA mode required adding a Postgres `LISTEN`/`NOTIFY` coordinator plus a relay mesh between replicas.

**Decision for 3.0: one active cao-server owner.** Workers dial a stable Service backed by a Deployment or StatefulSet configured with `replicas: 1`, plus #745's non-overlapping rollout/replacement procedure. The replica count alone is insufficient: default Deployment rolling updates may run an old and a new pod together, and a `ReadWriteOnce` volume is not a single-process lock. The current example uses a one-replica StatefulSet. Multi-replica hosting would need cross-replica routing/coordination and the other state changes in #775; it is not required for this release.

## Explicit non-goals

Keep the transport focused and integrate the contracts owned by the related tickets:

- **No mandatory message broker.** Reuse the existing local bus and direct remote channels; keep the laptop free of an additional service dependency.
- **No general-purpose at-least-once protocol for raw terminal output.** Bounded replay and visible gaps are sufficient for that stream. This does not remove #745's command acknowledgements, execution correlation, completion retention or cancellation-state requirements.
- **Not `workflow_run_event` as transport.** Its `seq` is per-run, allocated in memory, and it carries no destination or consumer cursor. It stays the post-ingestion audit history, which is a different job.
- **Sign-in and revocation policy belong to #774/#779, not a second authentication system here.** Authenticate and bind each connection to its permitted runtime from the first remote slice. Today's `X-CAO-Release-Token` is worker-specific and validated against that worker's Deployment (`examples/cao-clusters/kubernetes/eks/broker.py:1035-1079`, `:1265-1273`). Integrate restricted tenant/user delegation and live-channel revocation for 3.0; an arbitrary terminal ID or a previously accepted handshake is not continuing authorization. Keep the bounded runtime-control path available for authorized cancellation and final diagnostics.
- **No horizontal scaling of cao-server** — see the constraint above.

## Acceptance

- [ ] With cao-server and the agent in **separate pods**: live terminal viewing, input, and status all behave as they do in a single pod today.
- [ ] With cao-server and the agent in the **same pod or on a laptop**: behaviour and dependencies are unchanged.
- [ ] No new runtime dependency is required for local use.
- [ ] **In remote mode, nothing on the cao-server side calls tmux.** Status derivation, stale-pane probes, input injection, interactive attach, resize and output extraction run in the worker. The central server works in a container with **no tmux binary installed and no tmux socket reachable**; local mode retains its current execution path.
- [ ] The existing browser terminal preserves initial attachment, raw output, typing/paste, control keys, resizing, and disconnect-without-killing-the-agent against a remote worker. Exercise it separately from the background FIFO/status stream.
- [ ] Native CLI interactive attachment also works from a client without the runtime's local tmux/socket; detached launch retains its current meaning and behavior. Browser-only attachment does not satisfy the CLI requirement.
- [ ] Membership/role revocation and session expiry are enforced on already-open views, input operations and reconnects under #779. Closing a browser session is distinct from terminating an independently authorized agent; administrative cancellation still reaches the runtime after delegated user access is revoked.
- [ ] A status or completion message from a prior runtime/assignment generation cannot settle the current assignment. This does not establish prompt-turn freshness within that assignment (#735). Lost command responses do not cause blind duplicate work, and cancellation requested is not reported as execution stopped.
- [ ] **Gaps are visible rather than silent.** Each terminal's stream carries a monotonic position. A viewer that reconnects resumes from its last position instead of skipping what arrived while it was away, and a real gap is reported rather than absorbed — "the agent went quiet" stays distinguishable from "the text saying what went wrong was lost."
- [ ] **The connection handles disruption explicitly.** Use heartbeats/keepalive, automatic reconnect with backoff, and bounded replay or a visible gap/reset on resume. Authentication failure is surfaced rather than retried as an anonymous connection. Server restarts and rollouts must exercise this behavior without overlapping active owners (#745).
- [ ] Reconnect within the replay window, reconnect after eviction, a restarted stream generation, and queue overflow at the worker, server and browser boundary each produce the documented replay or visible gap. Include a dropped final chunk, not only a gap followed by another message.

Read #745 first, then trace the existing paths in services/fifo_reader.py, services/status_monitor.py, api/main.py, web/src/components/TerminalView.tsx, and cli/commands/launch.py. Map which operations touch the worker's tmux socket and how output, status, input, resize, replay, gaps, and acknowledgements cross the connection. Done means local behavior remains unchanged and remote browser and CLI sessions preserve output, status, input, sizing, reconnect, and detach behavior.

Written by the indexing model from the issue text.