Long-running handoff results can be lost after MCP timeout; make handoff durable and async

## Problem

Long-running agent work is common, but `handoff` currently makes the lifetime of
the work and its result depend on one blocking MCP tool call.

Observed with Codex:

```text
handoff(..., timeout=900)

Error: timed out awaiting tools/call after 600s
```

CAO accepts handoff execution timeouts up to 3600 seconds, while the shipped
Codex configuration gives MCP tool calls a 600-second default. The MCP client
can therefore stop waiting while the server-side worker continues.

Relevant code:

- [`providers/codex.py`](https://github.com/awslabs/cli-agent-orchestrator/blob/main/src/cli_agent_orchestrator/providers/codex.py#L300-L308) injects the 600-second default.
- [`mcp_server/server.py`](https://github.com/awslabs/cli-agent-orchestrator/blob/main/src/cli_agent_orchestrator/mcp_server/server.py#L692-L716) accepts the longer execution timeout and waits over synchronous HTTP for `timeout + 180`.
- [`services/agent_step.py`](https://github.com/awslabs/cli-agent-orchestrator/blob/main/src/cli_agent_orchestrator/services/agent_step.py#L228-L263) extracts a transient result and tears down the terminal on success.
- [`models/terminal.py`](https://github.com/awslabs/cli-agent-orchestrator/blob/main/src/cli_agent_orchestrator/models/terminal.py#L47-L60) explicitly defines `AgentStepResult` as transient.

The handoff timeout covers only the completion wait after the prompt is sent.
Provider initialization and a separate readiness wait happen first, so neither
the handoff value nor `timeout + 180` is a reliable total wall-clock budget.

## Why this is a correctness issue

If the MCP client times out first:

1. The worker may continue modifying its worktree.
2. The MCP process is blocked in synchronous `requests.post`, delaying
cancellation handling and follow-up MCP tools.
3. A late success can be extracted and the terminal deleted, but the abandoned
caller never receives the result.
4. If the worker reaches CAO's later execution timeout, the structured response
contains a live terminal ID, but the caller has already stopped waiting and
never receives it.
5. Retrying the handoff can start duplicate workers against the same task.

Delete-time logs and snapshots are useful forensic artifacts, but they are not
an authoritative, provider-extracted result with acknowledgement semantics.

## Desired behavior

Long-running execution should be asynchronous and server-owned. The supervisor
should not poll in an LLM-driven loop.

Suggested lifecycle:

1. Persist a queued handoff record and caller-supplied idempotency key.
2. Create the terminal and persist its ID as a separate state transition.
3. Return `handoff_id`, `terminal_id`, and state immediately.
4. Let CAO monitor execution independently of the MCP waiter.
5. Extract and persist the result before terminal teardown.
6. Publish an at-least-once completion hint to the supervisor inbox.
7. Let the supervisor retrieve and explicitly acknowledge the authoritative
result.
8. Retain unacknowledged results until a documented TTL expires.

The supervisor can dispatch other work and end its turn. Inbox delivery wakes
it when the handoff changes state. Status polling remains available for
recovery/operator inspection, but it is not the normal orchestration loop.

## Proposed tool contract

Either expose dedicated compatibility tools:

```text
handoff_start(...) -> {handoff_id, terminal_id, state}
handoff_status(handoff_id, wait_seconds=0)
handoff_result(handoff_id, acknowledge=true)
handoff_cancel(handoff_id)
```

or map native MCP Tasks (`tasks/get`, `tasks/result`, `tasks/cancel`, TTL and
status notifications) onto the same CAO durable task store when the client
negotiates support.

CAO should retain wrapper tools for clients without MCP Tasks support. There
must be one task state machine, not separate MCP and CAO implementations.

The existing `handoff` can remain as bounded synchronous convenience:

- return the current result shape when work finishes within the soft wait;
- return `state="running"` plus durable IDs before the provider transport
deadline when work continues;
- never turn running work into an ambiguous transport failure.

`assign` remains appropriate for persistent worker sessions and parallel work,
but reliable completion must not depend solely on the worker LLM remembering to
call `send_message`.

## Timeout model

Define separate budgets:

- provider/terminal initialization;
- post-creation readiness;
- worker execution after prompt submission;
- soft synchronous caller wait;
- HTTP/MCP transport watchdog.

Keep the synchronous wait around 600 seconds if desired, but set the transport
watchdog above it so CAO can return a structured `running` result. The worker
execution deadline may remain longer because it no longer owns the MCP call.

## Durability requirements

- Crash-consistent start and idempotent retry.
- Startup reconciliation of nonterminal handoff rows with live terminals and
backend windows.
- Atomic completion ordering: persist result/final state, then tear down.
- Compare-and-set handling for completion versus cancellation races.
- At-least-once, idempotent completion notifications keyed by `handoff_id`.
- Result acknowledgement separate from inbox delivery.
- Caller/session authorization for status, result, and cancel.
- Result size limits and documented retention/TTL behavior.
- Explicit cancellation semantics; cancelling an MCP waiter must not implicitly
cancel durable worker execution.

## Near-term fixes

Before the durable path lands:

1. Document `assign` as the required protocol for work likely to exceed the
synchronous budget.
2. Reject or soft-return handoff waits that exceed the effective provider MCP
budget instead of advertising an unreachable 3600-second blocking call.
3. Replace synchronous HTTP in the async MCP handler so one abandoned handoff
does not block all follow-up tools. This improves responsiveness but is not a
substitute for durable execution.
4. Fix Codex MCP timeout propagation: a `tool_timeout_sec` key directly in an
`mcpServers` config suppresses the 600-second default, but its supplied value
is not emitted. A later dotted `codexConfig` override currently works.

## Acceptance criteria

- A worker that outlives the MCP soft wait remains addressable by stable IDs.
- Its final extracted result is retrievable after caller timeout, MCP process
restart, and `cao-server` restart.
- Retrying with the same idempotency key does not create a second worker.
- Terminal teardown cannot happen before result persistence.
- Completion notification loss or duplication does not lose/duplicate the
authoritative result.
- Supervisor agents do not need `sleep` or status-poll loops.
- Cancellation, acknowledgement, authorization, and TTL behavior are tested.
- Integration coverage includes an MCP deadline shorter than worker execution,
real cancellation/disconnect, restart reconciliation, and completion/cancel
races.

## Related issues

- #291 overlaps on event-driven terminal waits and result capture, but does not
define durable ownership and acknowledgement of a handoff result after the
initiating MCP request is abandoned.
- #312 overlaps on durable orchestration/workflow concepts. A shared durable
task repository may support both, but the current handoff problem also affects
one-off long-running work.
- #317 addressed configurable timeout settings, but increasing a timeout alone
does not provide result durability, idempotency, or recovery.

Start with providers/codex.py, mcp_server/server.py, services/agent_step.py, and models/terminal.py to trace current timeout, result extraction, and teardown behavior. Review related issues #291 and #312, then use the acceptance criteria to define the durable handoff state transitions, recovery behavior, and integration coverage.

Written by the indexing model from the issue text.