Cross-Region resilience for AgentCore Runtime: synthetic canary detection and a failover-routing invoker

# Cross-Region resilience for AgentCore Runtime: synthetic canary detection and a failover-routing invoker
accompanying the upcoming [Disaster Recovery Resilience in Agentcore Developer Guide](https://preview.docs.aws.dev/preview/gillepa/IAD/AWSGenesisDocs/gillepa-dr-content/bedrock-agentcore/latest/devguide/disaster-recovery-resiliency.html):

## Summary

A deployable, end-to-end sample for **customer-driven cross-Region failover of AgentCore Runtime**. It covers the parts a customer must build themselves to fail Runtime traffic over between two Regions: a synthetic canary that health-checks the full invocation path, a CloudWatch alarm pipeline that turns the canary signal into a page, and an invoker that routes each request to a healthy Region with retry, circuit breaking, and session stickiness.

It is deployed as a warm standby across a Region pair (primary `us-west-2`, standby `us-east-1`), matching the design in the *Building custom disaster recovery for Amazon Bedrock AgentCore* topic in the AgentCore Developer Guide.

## Problem this fills

AgentCore Runtime is Region-scoped and has no native cross-Region failover. A customer who needs to keep an agent serving through a regional impairment has to build the detection-and-routing layer themselves. The repo already covers the **Memory** half of cross-Region DR — `01-features/04-manage-context-of-your-agent/memory/06-production-patterns/00-multi-region-replication/` (merged in #1747) replicates LTM and STM across Regions. What is missing is the **Runtime routing and health-detection** half: nothing in the repo shows how to detect that a Region is unhealthy and route invocations to the standby.

A search of the repo confirms the gap: `disaster recovery` returns no issues, and the `03-advanced` runtime folder's 13 samples (streaming, sessions, async, multi-agent, VPC, MCP, middleware, egress, async EC2 capacity provider) cover no multi-Region, failover, or health-driven routing concern.

## Scope boundary (what this is and is not)

**In scope:** synthetic canary agent + probe, CloudWatch alarm/SNS pipeline, and a failover-routing invoker (Region selection, retry with backoff, canary-synced circuit breaker, session-route stickiness, hard fallback). Route 53 failover-record configuration for the public endpoint.

**Out of scope, by design:** Memory replication. This sample assumes the Memory layer is already replicated and links to the existing `00-multi-region-replication` sample for it. The invoker carries the `actorId` and memory namespace on each call but does not itself move memory content.

## Differentiation from existing samples

| Existing | What it does | How this differs |
|---|---|---|
| `00-multi-region-replication` (memory, #1747) | Replicates LTM (Kinesis → Batch APIs) and STM (dual-write) across Regions | This routes **Runtime** invocations and detects Region health; it consumes the replicated Memory rather than producing it |
| `13-async-ec2-capacity-provider` | `@app.ping` `HealthyBusy`/`Healthy` for single-Region async instance reclamation | This uses an **external** canary across two Regions to drive **cross-Region** routing, not in-Region liveness |

## Proposed location (please confirm)

```
01-features/02-host-your-agent/01-runtime/03-advanced/14-cross-region-resilience/
```

Rationale: this is a cross-cutting Runtime concern, which is what `03-advanced` holds, and `14-` is the next index. It sits alongside the Runtime samples rather than under `memory/`, because the Memory replication sample already owns the data-layer concern. **Placement remains flexible based on maintainer preference** — if you would rather this live under `02-use-cases` or beside the memory sample, say so and I will adjust before opening the PR.

## Architecture

```
Amazon Route 53 (failover records + health checks)
│
┌────────────┴────────────┐
▼ ▼
Invoker (Lambda/ECS/lib) ── canary health (CloudWatch) ──┐
retry · circuit breaker · │
Region selection · session route │
│ │ │
┌───────────────┘ └───────────┐ │
▼ ▼ │
us-west-2 PRIMARY (ACTIVE) us-east-1 STANDBY │
┌───────────────────────┐ ┌───────────────────────┐│
│ AgentCore Runtime │ │ AgentCore Runtime ││
│ canary agent (CANARY_OK)│◄─ probe 1/min ─┐ │ canary agent │◄┘ probe 1/min
│ CanaryHealthy → CW │ │ │ CanaryHealthy → CW │
└───────────────────────┘ │ └───────────────────────┘
│ CW alarm (3×60s, TreatMissingData=breaching) → SNS → page
▼
AgentCore Memory (replica fed by the 00-multi-region-replication sample)
```

## Contents and structure

Adopts the `03-advanced` deploy/invoke/cleanup convention:

```
14-cross-region-resilience/
├── README.md six required sections
├── agent/
│ └── canary_agent.py minimal agent returning {"status":"CANARY_OK"} after the app path
├── canary/
│ └── canary_probe.py Lambda probe: invoke, validate sentinel, publish CanaryHealthy
├── infra/
│ └── template.yaml alarms, SNS, KMS-encrypted topic (per Region)
├── invoker/ deadline-bounded routing: circuit breaker, Region selection
├── deploy.py stand up both Regions: runtime, canary, alarms, invoker
├── invoke.py demonstrate a failover, not just a happy path
├── cleanup.py both Regions; flags the KMS key deletion waiting period
└── requirements.txt
```

## README outline (the six PR-checklist sections)

- **Introduction** — the Runtime failover problem, and the boundary with the Memory sample.
- **Architecture Diagram** — the diagram above, as an image.
- **Prerequisites** — two Regions with AgentCore Runtime; Python version; a replicated Memory (link the memory sample); IAM for `InvokeAgentRuntime` and CloudWatch in both Regions.
- **Usage** — `deploy.py`, then `invoke.py` to drive a happy path and a forced failover.
- **Sample Prompts** — prompts the canary agent and a demo agent answer, and the prompt used to show a failover mid-conversation.
- **Clean Up** — `cleanup.py`, with the KMS key deletion waiting period called out.

## Acceptance criteria

- Deploys cleanly into a two-Region pair via `deploy.py`.
- `invoke.py` demonstrates: a healthy invocation, a forced-unhealthy primary, and a routed-to-standby invocation within the client deadline.
- Passes the repo's `checkov` and `secrets` gates.
- `cleanup.py` removes every resource in both Regions.

## Open question for maintainers

1. Do you agree with the proposed location, or prefer a different folder?
2. Any objection to referencing (not duplicating) the `00-multi-region-replication` sample for the Memory layer?

Start by reading the existing 03-advanced Runtime samples and the 00-multi-region-replication memory sample to verify conventions and boundaries. Then map the proposed 14-cross-region-resilience structure, including agent/canary, infra/template.yaml, deploy.py, invoke.py, cleanup.py, and invoker. Done means both Regions deploy, forced failover reaches the standby within the deadline, checkov and secrets gates pass, and cleanup removes both Regions' resources.

Written by the indexing model from the issue text.