awslabs / awslabs/cli-agent-orchestrator

[Feat] External orchestration driver: event-driven waits, worker governance, git-based result contract (implementation available)

Open
#291 0 comments 0 reactions 0 assignees View on GitHub
feature
Dominant language
Python
Stars
1.3k
Forks
267
Avg merge
1d 23h
Merged PRs (30d)
70

Description

## Overview
I've been using CAO as an **orchestration backend**: a primary agent (running in its own chat loop) drives CAO through `cao-ops-mcp` to spawn worker agents (Claude Code / Kimi), delegate coding tasks, review the results, and merge. Doing this on a real project surfaced a set of gaps, and I ended up implementing all of them on a branch. Filing this as a feature suggestion + offer to upstream.

Branch with the full implementation (26 commits on top of the #273 base, each independently scoped, unit + live e2e tested): **https://github.com/call-me-ram/cli-agent-orchestrator/tree/feat/ops-mcp-orchestration-driver**

## What the branch adds

**1. Event-driven waits — no polling anywhere in the driver path**
- `GET /terminals/{id}/wait?status=completed&status=error&timeout=600` — long-poll that parks on the internal EventBus and returns the instant the status transition is published. Timeout capped (3600s), client-disconnect watched, lap-based re-snapshot so it also resolves on backends that never publish status events.
- `GET /events` — SSE stream of all terminal status changes (heartbeats, stalled-client teardown, leak-proof unsubscribe, same client allowlist as the terminal WebSocket).
- `cao-ops-mcp` gains `wait_for_terminal_status` (blocks server-side; sync HTTP runs via `asyncio.to_thread` so the MCP event loop stays responsive) plus `get_terminal_output` / `get_terminal_status` / `get_terminal_result` tools; `get_session_info` returns live per-terminal status.
- `handoff`/`assign` in `cao-mcp-server` use the same event-driven wait, treat ERROR as a terminal outcome (fail fast instead of burning the timeout), and a timed-out handoff now interrupts the runaway worker (Escape) and attaches partial output + git state instead of abandoning it.

**2. Worker governance**
- Concurrency caps: `CAO_MAX_CONCURRENT_WORKERS` (global) and `CAO_MAX_WORKERS_PER_SESSION`, enforced before any backend resource is created, counting only live workers (passive provider-registry check — stale DB rows are invisible). Cap rejection maps to HTTP 429 and handoff/assign surface a structured back-off message.
- Per-worker **git worktree isolation** (`CAO_ENABLE_GIT_WORKTREE`): each worker gets its own worktree + branch (`cao/-`), so parallel workers can never collide on files. Removal auto-snapshots uncommitted/untracked work onto the worker's branch first — handoff auto-delete can no longer destroy output. Repo-subdirectory launches land in the same subdirectory of the worktree; kimi (temp-dir cwd) excluded.
- Network egress gating: `web_fetch` CAO tool maps `WebFetch`/`WebSearch` (claude) and `web_fetch`/`google_web_search` (gemini), so read-only reviewer/supervisor roles have no egress channel; launch-time warning when a restricted policy targets a soft-enforcement provider (kimi/codex).

**3. Git-based result contract (review real diffs, not scraped TUI text)**
- `GET /terminals/{id}/result` + `terminal_service.get_result()`: branch, changed files, diff vs HEAD (capped), **untracked file contents** (new modules are invisible to `git diff` alone), `shared_working_directory` flag, per-terminal `.cao/result-.json` manifest. `HandoffResult` carries it, captured before worker cleanup.

**4. Reliability under restarts**
- On startup the server **re-attaches to surviving terminals** (tmux outlives the process): providers re-created from DB metadata, FIFO readers restarted, pipe-pane re-pointed, and an initial status seeded from the rendered pane so silent (idle/finished) agents don't read UNKNOWN forever. Observed live before the fix: a supervisor's handoff blocked toward its full timeout because the restarted server couldn't see its finished worker.
- `X-Server-Time` response header so UIs can correct server/browser clock skew (WSL2 drift of 5.5h observed — relative timestamps were nonsense).

**5. #287 implemented: pyte rendered-screen status detection, now default-on**
- Per-terminal `pyte.Screen` in StatusMonitor with rising-edge/quiescence debounce (naive per-chunk rendering measured WORSE than the raw path — the debounce is load-bearing), provider opt-in via `supports_screen_detection` (claude_code, kimi_cli), `CAO_PYTE_STATUS=false` falls back to the raw path. Validated live (init, multi-turn, send_message, handoff on real agents) and by the full e2e gauntlet.

## Verification
~2,350 unit tests green on the branch; full e2e gauntlet (allowed-tools, assign, cross-provider, handoff, send_message, skills, supervisor orchestration on live claude + kimi) green in pyte mode; live driver validation (event wait resolves at the exact transition, 11ms already-reached path).

Happy to split this into a sequence of reviewable PRs (the commits are already scoped that way) if there's interest — suggested order: event-driven waits/SSE → result contract → caps/worktrees → recovery → pyte default.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the implementation branch and its scoped commits, beginning with the event-driven wait and SSE entry points in cao-ops-mcp and cao-mcp-server. Then trace terminal_service.get_result() and StatusMonitor, and run the unit and live e2e verification described in the issue. Done means the listed waits, governance, result contract, restart recovery, and pyte detection work together without regressions.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
api, backend, devtools, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.