aws-samples / aws-samples/sample-apj-sup-sa

Proposal: add bedrock-observability-dashboard sample under ai-infra/

Open
#91 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
19
Forks
30
Avg merge
23h 54m
Merged PRs (30d)
28

Description

Proposing a new sample for the **ai-infra/** domain, per CONTRIBUTING (opening an issue to discuss significant work before a PR).

## What it is
A CloudWatch **observability dashboard for Amazon Bedrock** foundation-model usage (`InvokeModel` / `Converse` / `ConverseStream` / `InvokeModelWithResponseStream`), delivered as a **single tiered CloudFormation template**. Built entirely from native AWS data sources — no third-party components.

## Why it fits ai-infra/
The folder covers SageMaker, fine-tuning, inference, and training. This adds the **inference observability** angle for Bedrock — complementary to the existing EKS-based `finetuning-on-eks` and `robotics-foundation-models-on-eks` samples. Many startups running on Bedrock ask exactly this: "how do I see latency, errors, token usage, and per-team attribution without standing up third-party tooling?"

## Highlights
- **Tiered, parameter-gated design** — one template, three additive tiers selected by CloudFormation `Conditions`; an empty parameter means the resource is never created; tiers upgrade in place (same dashboard URL).
- **Tier 1:** invocations, latency (p50/p90/p99), errors/throttles, per-model token usage (input/output + cache read/write), quota headroom — CloudWatch metrics only.
- **Tier 2:** per-tenant attribution + saved Logs Insights queries via Bedrock Model Invocation Logs.
- **Tier 3:** IAM identity, error reasons, cross-region routing via CloudTrail management events.
- **Region-agnostic per-model widgets** via CloudWatch `SEARCH` on the `ModelId` dimension — cross-region inference prefixes (`us.`/`eu.`/`apac.`) work with no template edits; any invoked model appears automatically.
- **Supporting assets** — architecture/setup/IAM docs, 8 standalone Logs Insights queries, an optional CloudTrail→CloudWatch-Logs prerequisite template, and a 24h synthetic workload generator for demos.

## Quality / conventions
- **Tests:** `python3 -m pytest` → 22 passed (per-tier structure, metric names, query/template sync, tier gating; renders the dashboard body to valid JSON for tiers 1–3).
- **`cfn-lint`:** 0 errors on both templates.
- Follows the existing `ai-infra/` sub-project layout (project-local `README.md` + `.gitignore`, no duplicated root boilerplate); updates `ai-infra/README.md` with the new row.
- ~37 files, MIT-0, no secrets or real account IDs.

## Scope note
This sample covers **runtime InvokeModel/Converse observability only**. Bedrock Agents, Guardrails standalone metrics, Knowledge Bases, and Provisioned Throughput utilization are explicitly out of scope (documented as such).

Branch is ready (`FilledVaccum:feat/bedrock-observability-dashboard`). Happy to open the PR once a maintainer is OK with the direction. Thanks!

Contributor guide

Open the contributing guide

Research direction

Start by reading CONTRIBUTING and the existing ai-infra/ project layout, then inspect the proposed FilledVaccum:feat/bedrock-observability-dashboard branch. Run python3 -m pytest and cfn-lint on both templates; done means the 22 tests pass, both templates lint cleanly, and ai-infra/README.md includes the new sample.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
ai-infra-agents, cloud, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.