aws-samples / aws-samples/sample-apj-sup-sa
Proposal: add bedrock-observability-dashboard sample under ai-infra/
- Dominant language
- Python
- Stars
- 19
- Forks
- 30
- Avg merge
- 23h 54m
- Merged PRs (30d)
- 28
Description
Proposing a new sample for the **ai-infra/** domain, per CONTRIBUTING (opening an issue to discuss significant work before a PR).
## What it is
A CloudWatch **observability dashboard for Amazon Bedrock** foundation-model usage (`InvokeModel` / `Converse` / `ConverseStream` / `InvokeModelWithResponseStream`), delivered as a **single tiered CloudFormation template**. Built entirely from native AWS data sources — no third-party components.
## Why it fits ai-infra/
The folder covers SageMaker, fine-tuning, inference, and training. This adds the **inference observability** angle for Bedrock — complementary to the existing EKS-based `finetuning-on-eks` and `robotics-foundation-models-on-eks` samples. Many startups running on Bedrock ask exactly this: "how do I see latency, errors, token usage, and per-team attribution without standing up third-party tooling?"
## Highlights
- **Tiered, parameter-gated design** — one template, three additive tiers selected by CloudFormation `Conditions`; an empty parameter means the resource is never created; tiers upgrade in place (same dashboard URL).
- **Tier 1:** invocations, latency (p50/p90/p99), errors/throttles, per-model token usage (input/output + cache read/write), quota headroom — CloudWatch metrics only.
- **Tier 2:** per-tenant attribution + saved Logs Insights queries via Bedrock Model Invocation Logs.
- **Tier 3:** IAM identity, error reasons, cross-region routing via CloudTrail management events.
- **Region-agnostic per-model widgets** via CloudWatch `SEARCH` on the `ModelId` dimension — cross-region inference prefixes (`us.`/`eu.`/`apac.`) work with no template edits; any invoked model appears automatically.
- **Supporting assets** — architecture/setup/IAM docs, 8 standalone Logs Insights queries, an optional CloudTrail→CloudWatch-Logs prerequisite template, and a 24h synthetic workload generator for demos.
## Quality / conventions
- **Tests:** `python3 -m pytest` → 22 passed (per-tier structure, metric names, query/template sync, tier gating; renders the dashboard body to valid JSON for tiers 1–3).
- **`cfn-lint`:** 0 errors on both templates.
- Follows the existing `ai-infra/` sub-project layout (project-local `README.md` + `.gitignore`, no duplicated root boilerplate); updates `ai-infra/README.md` with the new row.
- ~37 files, MIT-0, no secrets or real account IDs.
## Scope note
This sample covers **runtime InvokeModel/Converse observability only**. Bedrock Agents, Guardrails standalone metrics, Knowledge Bases, and Provisioned Throughput utilization are explicitly out of scope (documented as such).
Branch is ready (`FilledVaccum:feat/bedrock-observability-dashboard`). Happy to open the PR once a maintainer is OK with the direction. Thanks!
Contributor guide
Research direction
Start by reading CONTRIBUTING and the existing ai-infra/ project layout, then inspect the proposed FilledVaccum:feat/bedrock-observability-dashboard branch. Run python3 -m pytest and cfn-lint on both templates; done means the 22 tests pass, both templates lint cleanly, and ai-infra/README.md includes the new sample.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- ai-infra-agents, cloud, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100