awslabs / awslabs/cli-agent-orchestrator

[Kiro CLI] v3/KAS compatibility and migration plan

Open
#443 1 comment 1 reaction 1 assignee Claimed by @fanhongy View on GitHub
enhancement feature
Dominant language
Python
Stars
1.3k
Forks
267
Avg merge
1d 23h
Merged PRs (30d)
70

Description

## Overview

CAO's Kiro provider currently depends on Kiro CLI v2 behavior but does not explicitly select an engine. It launches `kiro-cli chat --agent `, installs a v2-style JSON custom-agent profile, and infers lifecycle state from the current terminal UI.

Kiro CLI v3/KAS changes the agent profile and permission model, MCP startup behavior, and parts of the CLI/TUI surface. We should support v2 and v3 side by side, keep v2 as CAO's default during migration, and make v3 the default only after the acceptance gates below pass.

This is also a security migration. CAO must not report restricted KAS sessions as supported until its `allowedTools` and `toolsSettings` policy can be represented and verified with KAS Cedar permissions.

## User stories

- As an existing Kiro v2 user, I can upgrade CAO without an implicit engine switch or profile regression.
- As an early adopter, I can explicitly select KAS and get clear diagnostics when my installed CLI or profile is incompatible.
- As a CAO operator, I get the same effective tool restrictions and orchestration guarantees on either supported engine.
- As a maintainer, I can test and roll back each engine independently.

## Confirmed blockers

The following follow from CAO's current implementation and the reviewed KAS implementation:

- CAO does not pass an engine selector in `src/cli_agent_orchestrator/providers/kiro_cli.py`; behavior can therefore change when Kiro changes its own default.
- `src/cli_agent_orchestrator/services/install_service.py` and `src/cli_agent_orchestrator/models/kiro_agent.py` always emit the v2 JSON shape, including `allowedTools`, `toolsSettings`, and inline `mcpServers`.
- The KAS custom-agent loader implementation snapshot reviewed in the supplied PDF skips CAO-generated profiles that use CAO's current v2 schema. In that reviewed snapshot, the loader strips legacy `allowedTools` / `toolsSettings` fields and requires KAS-compatible profile data and Cedar permissions instead. This finding is specific to the reviewed loader snapshot; the released wrapper and future KAS versions still require capability and profile-loading smoke tests.
- CAO's status detection, response extraction, approval detection, startup fallback, and exit flow are coupled to observed v2/current-TUI output in `src/cli_agent_orchestrator/providers/kiro_cli.py` and `src/cli_agent_orchestrator/services/herdr_inbox_service.py`.
- Current Kiro tests and fixtures under `test/providers/` and `test/e2e/` do not form an explicit v2/KAS matrix.

These are sufficient reasons not to make KAS the CAO default yet.

## Unknowns requiring authenticated KAS smoke tests

Do not treat these as confirmed breakages until they are exercised against the released, authenticated CLI wrapper:

- The exact support and precedence of `--agent-engine`, `--v3`, `--agent`, `--legacy-ui`, `--trust-all-tools`, and `--model` across supported Kiro CLI versions.
- The released wrapper's custom-agent discovery rules and the exact v3 profile artifact accepted by `chat --agent`.
- Whether profile-scoped `cao-mcp-server` configuration loads under KAS, how its tool names are exposed, and when it becomes ready relative to the first idle prompt.
- KAS terminal output for startup, idle, processing, completion, errors, permission approval, response extraction, and `/exit` in tmux/headless operation.
- Whether `.kiro/steering/cao-memory.md` is loaded with the required inclusion and workspace-trust behavior in a CAO-launched KAS session.
- Model selection behavior and failure messages for unavailable or engine-specific model IDs.

Capture sanitized terminal fixtures and version/capability results from these tests so behavior is not inferred from wrapper internals alone.

## Proposed implementation

### Phase 0: Pin v2 and introduce engine capabilities

- [ ] Add an explicit Kiro engine setting with at least `v2` and `kas`; keep `v2` as the default.
- [ ] Pass the selected engine on every Kiro launch instead of inheriting the Kiro CLI default.
- [ ] Detect CLI version and probe required capabilities/flags. Do not infer all capabilities from a version string alone.
- [ ] Reject unsupported engine/flag combinations before creating a terminal, with an actionable error.
- [ ] Record the selected engine in terminal/session metadata and diagnostics.
- [ ] Keep engine selection available through the CLI and API paths used by launch, handoff, and assign.

### Phase 1: Generate engine-aware profiles and permissions

- [ ] Split the current Kiro profile renderer into v2 and KAS adapters while retaining a shared CAO profile model.
- [ ] Continue generating the existing JSON artifact for v2.
- [ ] Generate and validate the KAS-supported custom-agent format without overwriting the v2 artifact.
- [ ] Compile KAS tool visibility/availability and Cedar permissions/approval behavior from CAO's resolved `role`, `allowedTools`, `toolsSettings`, and explicit deny policy.
- [ ] Define mappings by capability, including filesystem read/write, shell execution, web access, skills/subagents, and MCP.
- [ ] Treat every non-wildcard CAO capability not granted by the resolved profile as denied, not merely "ask".
- [ ] Abort install/launch if any restriction cannot be represented safely. Never silently omit or weaken a rule.
- [ ] Preserve wildcard/unrestricted behavior only when the user explicitly selected `allowedTools: ["*"]` or CAO's equivalent `--yolo` flow.
- [ ] Add a profile lint/migration command that reports unsupported fields and the resulting v2 and KAS policies.

### Phase 2: Restore MCP and provider lifecycle parity

- [ ] Render profile-scoped `cao-mcp-server` configuration in the KAS-supported form.
- [ ] Add an MCP readiness gate: if a profile requires `cao-mcp-server`, do not mark the terminal ready for orchestration until the expected server and tools are discoverable.
- [ ] Surface MCP startup failures and timeouts distinctly from model/TUI startup failures.
- [ ] Make launch arguments and TUI parsing engine-aware, covering startup, idle, processing, completion, errors, approvals, and response extraction.
- [ ] Verify both default TUI and `--legacy-ui` where the selected engine advertises support.
- [ ] Verify `--trust-all-tools` behavior without bypassing the translated Cedar restrictions accidentally; separate "skip approvals" from "grant capabilities".
- [ ] Verify model selection and produce a clear error when a configured model is not supported by the selected engine.
- [ ] Verify graceful `/exit`, timeout recovery, and cleanup for each engine.
- [ ] Verify CAO memory injection through `.kiro/steering/cao-memory.md`, including workspace trust, update visibility, and no cross-workspace leakage.

KAS's internal sub-agent implementation is separate from CAO's orchestration model. CAO launches independent terminal processes and coordinates them through `cao-mcp-server`; KAS's internal sub-agent changes do not inherently replace or break handoff/assign/send-message flows as long as the profile-scoped CAO MCP server loads and is ready.

### Phase 3: Build the dual-engine test matrix

- [ ] Unit-test engine selection, capability detection, launch arguments, profile rendering, Cedar translation, and fail-closed errors.
- [ ] Add separate sanitized v2 and KAS terminal fixtures for every provider state and response-extraction path.
- [ ] Integration-test profile installation and loading for both engines, including restricted, unrestricted, model-pinned, and profile-scoped MCP configurations.
- [ ] Run authenticated E2E coverage for both engines: launch/profile load, restrictions, approval handling, memory/steering, handoff, assign, callback `send_message`, TUI state transitions, response extraction, and exit.
- [ ] Test MCP slow-start, unavailable-server, malformed-profile, unsupported-flag, invalid-model, and CLI-upgrade scenarios.
- [ ] Keep the v2 and KAS E2E jobs independently selectable so a KAS failure does not hide a v2 regression.

Relevant existing coverage includes:

- `test/providers/test_kiro_cli_unit.py`
- `test/providers/test_kiro_cli_integration.py`
- `test/providers/fixtures/kiro_cli_*`
- `test/e2e/test_allowed_tools.py`
- `test/e2e/test_handoff.py`
- `test/e2e/test_assign.py`
- `test/e2e/test_send_message.py`
- `test/plugins/builtin/test_kiro_cli_memory.py`

### Phase 4: Documentation and rollout

- [ ] Document engine selection, supported Kiro CLI versions/capabilities, profile formats, security semantics, and troubleshooting in `docs/kiro-cli.md`.
- [ ] Update `docs/tool-restrictions.md` only after hard enforcement has been demonstrated on KAS.
- [ ] Release KAS as opt-in and gather compatibility results before proposing a default change.
- [ ] Publish any default-change proposal with a migration window and release-note callout.
- [ ] Do not deprecate v2 until KAS has passed the gates below for at least one stable CAO release and Kiro's own v2 support horizon is known.

## Security requirements

- Restricted CAO profiles must remain hard-restricted under KAS. A prompt, approval choice, alias, shell indirection, internal sub-agent, or MCP tool must not regain a denied capability.
- Cedar generation must be deterministic, reviewable in diagnostics, and covered by allow/deny tests. Deny must take precedence.
- Missing, malformed, or partially translated policy must fail closed before the agent can process a task.
- Workspace trust and protected-path behavior must be tested explicitly; no test should read real credentials or depend on a developer's personal Kiro configuration.
- CAO must not equate MCP server presence with permission to use every MCP capability. The resolved profile remains the source of policy.
- Engine fallback must never silently change the effective security policy.

## Acceptance criteria for making KAS the default

- [ ] Every Kiro launch explicitly selects an engine, and v2 remains behaviorally compatible with the current supported flow.
- [ ] A CAO-generated KAS profile is selected by name and its prompt, model, resources, and MCP configuration are demonstrably active.
- [ ] For every resolved CAO `role` / `allowedTools` / `toolsSettings` policy, the KAS profile compiler deterministically derives both the tools visible/available to KAS and the corresponding Cedar permissions and approval behavior; tests verify both outputs against the resolved policy.
- [ ] Restricted profiles block every denied capability in authenticated E2E tests; allowed capabilities work; explicit unrestricted mode is tested separately.
- [ ] `cao-mcp-server` readiness is verified before CAO reports an orchestration-capable terminal as ready.
- [ ] Handoff, assign, callback `send_message`, inbox delivery, and terminal exit pass repeatedly on both engines.
- [ ] Startup, state detection, response extraction, approvals, `--legacy-ui`, `--trust-all-tools`, model selection, and `/exit` have documented outcomes for the supported released wrapper.
- [ ] Steering/memory injection is present in KAS model context and remains workspace-scoped.
- [ ] Unit, fixture, integration, and authenticated E2E suites run as an explicit v2/KAS matrix in CI or the documented release qualification process.
- [ ] Upgrade and rollback instructions are published, with no destructive conversion of existing v2 profiles.
- [ ] No open P0/P1 security or orchestration regression remains.

## Rollback behavior

- Preserve v2 profile artifacts and the explicit v2 launch path throughout the migration.
- Before KAS becomes default, a KAS failure should fail with diagnostics and require an explicit retry on v2; it must not silently relaunch with different permissions.
- If KAS later becomes default, retain a documented configuration/CLI override and release-level kill switch to restore v2 without reinstalling profiles.
- A rollback must keep terminal/session metadata truthful and must not reuse a partially initialized KAS terminal as v2.

## Relevant code

- `src/cli_agent_orchestrator/providers/kiro_cli.py`
- `src/cli_agent_orchestrator/services/install_service.py`
- `src/cli_agent_orchestrator/models/kiro_agent.py`
- `src/cli_agent_orchestrator/services/status_monitor.py`
- `src/cli_agent_orchestrator/services/herdr_inbox_service.py`
- `src/cli_agent_orchestrator/plugins/builtin/kiro_cli_memory.py`
- `src/cli_agent_orchestrator/utils/tool_mapping.py`
- `docs/kiro-cli.md`
- `docs/tool-restrictions.md`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.