[Feat] Durable run record: key a handoff's submission and result, not just its terminal

## Summary

> **Status update.** PR #634 has since REMOVED its `--idempotency-key` flag rather than
> shipping it with a documented caveat. So this issue is no longer "fix a shipped flag" —
> it is **build the run record, then expose the keyed retry**. There is no unsafe
> user-facing surface in the meantime. #634 retains the whole server-side substrate
> (`idempotency_keys` table, request fingerprinting, the 409 conflict arm, key-derived
> session names, the derived memory-manager sidecar key, the retention sweep) plus an
> unreachable-but-tested `idempotency_key` parameter on `_handoff_impl`, specifically so
> this work has a seam to build on.
>
> **#616 stays open pending this issue.** #634's body was changed from `Fixes #616` to
> `Refs #616` for that reason: #616 requires that "killing the CLI process after submission
> does not lose the job or its result", and its proposal asks for
> `cao agent status|result|cancel JOB_ID`. Both are this run record. Closing this issue is
> what closes #616.

Two open PRs each defer one half of the same missing primitive: a **durable run record** that ties a caller-supplied request key to what actually happened to the run. Neither half can be closed inside the PR that found it, and both need the same substrate, so they are tracked here together rather than as two issues.

- **PR #634** owns retry-safe request IDs for terminal **creation** — the `idempotency_keys` table mapping a caller-supplied key to the terminal it created.
- **PR #453** owns durable **result** storage — the `handoff_results` table and its retrieval path.
- The gap is what sits between them: nothing records *what happened to the run* under that key, so a retry cannot tell a submitted-and-finished run from one that never started.

#636 was closed as completed on 2026-08-18 while both halves were still open. This issue is not a duplicate of it — it is the remainder it left behind.

## Deferral 1 — a keyed handoff retry re-sends the task (from PR #634)

Reported by @haofeif: https://github.com/awslabs/cli-agent-orchestrator/pull/634#discussion_r3888433568

As reported, a key hit returned the terminal the first attempt created while every invocation still reached the submission, so the prompt was delivered a second time. The flag that exposed this is now gone; the behaviour description stands as the requirement this issue has to satisfy before any keyed retry is exposed again.

The reason this cannot be fixed inside #634 is sharper than a scope boundary. **Nothing anywhere records whether the first attempt's message actually landed.** So on a retry:

- **skipping** the send can silently DROP the task — if the first attempt died between create and send, the worker exists and is idle forever, and the caller is told it reattached;
- **re-sending** DUPLICATES it — if the first attempt did deliver, the worker now gets the same task twice.

Both available answers are wrong in one direction, and no amount of care inside `_handoff_impl` can pick correctly, because the information needed to decide is not written down. That is the whole argument for a run record, and it is why #634 stopped at the creation half instead of guessing.

What #634 shipped, and what it deliberately did not:

- keyed creation is deduplicated: `services/terminal_service.py:652` `create_terminal`, key resolution at `:911`, request fingerprint at `:453`
- storage and sweep: `clients/database.py:328` `IdempotencyKeyModel`, `:1339` `get_idempotency_record`, `:1365` `delete_idempotency_key`, TTL sweep in `services/cleanup_service.py:79`
- the submission is NOT keyed: `utils/orchestration.py:1226` `_send_direct_input_handoff` on the `--no-wait` path, and `:1183` / `:1270` `_run_step_and_build_result` on the blocking paths — all reached unconditionally on a key hit

There is no `--idempotency-key` flag any more, so nothing user-facing promises a safety property it cannot hold. `cli/commands/agent.py` now documents only the MANUAL recovery route (the `terminal_id` printed to stderr before the wait) and points here for the automatic one.

## Deferral 2 — `job_id` never reaches the caller in #447's actual ordering (from PR #453)

Reported by @call-me-ram: https://github.com/awslabs/cli-agent-orchestrator/pull/453#pullrequestreview-4723835945, still open at https://github.com/awslabs/cli-agent-orchestrator/pull/453#pullrequestreview-4784240377

`job_id` reaches the supervisor through exactly one carrier: the `HandoffResult` returned from the `except requests.Timeout` branch. That branch fires on the *internal* HTTP deadline (`timeout + 180s`). The reported failure is the opposite ordering:

- internal client timeout: e.g. `900 + 180 = 1080s`
- provider `tools/call` deadline: e.g. Codex's `600s`

The provider transport dies at 600s while the call is still parked in `requests.post`. The timeout branch never runs, nothing is returned, and the durably persisted row is undiscoverable — there is no listing or lookup-by-terminal endpoint, and the id is minted per call and never logged.

Note for whoever picks this up: #453's review quotes `server.py:725` / `server.py:766-776`, but PR #634 extracted that code into `utils/orchestration.py`. The current homes are `_run_step_and_build_result` (`utils/orchestration.py:866`) and `_handoff_impl` (`:997`). Read the #453 anchors against its own head, not against main after #634 lands.

## Why one issue

These are the same missing primitive seen from two directions. #634 can tell you *which terminal* a key created; #453 can tell you *what a finished run produced*. Neither can tell you **whether the run under this key was ever submitted, and if so what became of it** — which is exactly what both deferrals need. Fixing them separately would mean building the run record twice, or building it once and retrofitting the other half onto it anyway.

A caller-supplied key already exists and is already durably committed in the same transaction as the terminal row, so the identity half of the problem is solved. What is missing is state transitions recorded against that key.

## Suggested shape (not prescriptive)

Extend the keyed record from "key -> terminal" to "key -> run", carrying at minimum a state (`created` / `submitted` / `completed` / `errored`) and the extracted result once there is one. That makes both deferrals fall out:

- a retry reads the state and either reattaches without re-sending (`submitted`), returns the stored result (`completed`), or sends for the first time (`created`) — no guessing
- the key doubles as the discoverable handle #447 needs, so a caller who lost the response can look up the run by a value it chose itself, rather than by a `job_id` it never received

Whether this extends `idempotency_keys`, extends `handoff_results`, or introduces a third table is an open design question. Worth settling deliberately, since it is the seam both PRs will build on.

## Constraints worth preserving

- The unkeyed path must stay byte-for-byte unchanged — both PRs kept their features opt-in and default-off, and that is what made them reviewable.
- `idempotency_keys` is **node-local** SQLite. A keyed retry that lands on a different `target_host` cannot dedupe by construction. PR #634 rejects `target_host` together with `--idempotency-key` / `--no-wait` for this reason; any run record needs the same treatment or an explicit cross-node story.
- Rows need a retention story. `idempotency_keys` has a TTL sweep in `cleanup_service`; a run record carrying result payloads will need one at least as aggressive.
- Do not persist or log task text in plaintext. #634's fingerprint hashes the delivered message into a sha256 digest specifically to avoid this.

## References

- PR #634 — https://github.com/awslabs/cli-agent-orchestrator/pull/634 (issue #616)
- PR #453 — https://github.com/awslabs/cli-agent-orchestrator/pull/453 (issue #447)
- #636 — closed as completed 2026-08-18, before either half landed

Start with the keyed-record code in clients/database.py and the orchestration entry points utils/orchestration.py:_handoff_impl and _run_step_and_build_result; compare them with the referenced PR #634 and PR #453 behavior. Decide deliberately how the run record extends the existing storage, including retention and node-local constraints. Done means keyed retries distinguish created, submitted, completed, and errored runs, expose a lost run through its key, and leave the unkeyed path unchanged.

Written by the indexing model from the issue text.