[meta] Operational observability for the SDLC pipeline (operator-facing)

> **This is a meta / planning issue, not a triggerable one.** It lays out the architecture for operator-facing observability over the SDLC pipeline and decomposes the work into separate implementation issues. **Do NOT add `agent:start` to this issue.** The agent should not act on it directly.
>
> Each numbered deliverable below corresponds to a separate child issue (links added as they're filed).

## Audience

Enterprise team **operating** the SDLC pipeline — i.e., on-call engineers and platform maintainers who need to debug stuck or failed pipeline runs, monitor pipeline health, and respond to alerts. Distinct from end users (issue authors), whose visibility is via the issue's labels and comments today and is acceptable as-is.

## Problem

A maintainer debugging a stuck pipeline today must manually correlate several disconnected sources:

- The GitHub issue page (label state, agent comments).
- AWS Step Functions console (per-execution status, Lambda invocation outcomes).
- Multiple CloudWatch log groups: Setup Lambda, Pipeline Lambda, the AgentCore runtime's `otel-rt-logs` stream (claude OTel events), the runtime's `[runtime-logs]` streams (FastAPI + microVM stdout).
- The runtime itself, via `bedrock-agentcore:InvokeAgentRuntimeCommand` probes (the `test-runtime-coding-agent` skill exists exactly because this is recurring work).
- The orchestrator's local artifacts at `/mnt/workplace/gitproject/.dev-claude/invocation-N/{explore,feedback,critique,pr}.md` — invisible from outside the runtime today.

This stitching cost was demonstrated repeatedly during the 2026-06-03/04 debugging cycle for issues #23 and #27. Several real failure modes (oversized `issue.json`, conversation state on rootfs, stale-base local commits, hallucinated 'PR merged' exits) only became visible after multi-step manual probing.

There is no real-time monitoring, no alerting, no metrics dashboard, no per-run trace, no centralized error reporting. A pipeline can fail silently overnight and nobody notices until the next time a human looks at the issue.

## Out of scope (covered by other tracks)

- **End-user-facing observability** — the issue label state machine + agent comments are sufficient. Don't conflate this with operator observability.
- **Inspector POC for live session listing (#24)** — a different path. The inspector is a session-list UI; this is metrics + logs + alerts. Independent tracks; can ship in parallel.
- **Workflow / orchestrator skill changes** — out of scope. This is about *visibility into* what the pipeline does, not changing what it does.
- **AgentCore Observability built-in dashboard** — already documented in `coding-assistants/claude-code/obs.md`. We use it where it's sufficient; this issue covers what it doesn't cover (cross-Lambda traces, alarms, dashboards purpose-built for the SDLC pipeline shape).

## Proposed architecture

Three independent observability components. Each is a separate child issue:

### A. Structured Pipeline Run Trace (\"per-run timeline\")

A single source of truth that, given an issue number or session ID, produces an ordered timeline of:

1. Step Functions execution start / Lambda invocations / Lambda outcomes.
2. Setup Lambda log lines: `[github-setup]`, `[refresh_for_reinvocation]`, `[github-setup] stop_runtime_session non-fatal error (continuing)`, etc.
3. Pipeline Lambda log lines.
4. AgentCore runtime claude OTel events (`api_request`, `tool_result`, `tool_decision`, `mcp_server_connection`).
5. `.dev-claude/invocation-N/` artifacts (file paths and sizes; not contents — those are big).
6. Issue label transitions (read from GitHub API).
7. PR creation / push events (read from GitHub API).

**Implementation options to evaluate (the implementation issue picks one):**

- **A1: CloudWatch Logs Insights pre-built saved queries.** Store the queries from `obs.md` as named saved queries in CloudWatch (resourced via CDK). Operators run them via the console. Cheap, but still requires manual stitching across log groups.
- **A2: A small Lambda (`pipeline-trace-lambda`) that, given an `issue_number` or `session_id`, joins all sources and returns a structured JSON timeline.** Invoked manually via CLI or via the inspector POC. Medium effort.
- **A3: Step Functions execution itself emits structured events to an EventBridge bus**, all consumers (Lambda log lines, claude OTel events) tagged with the SFN execution ARN, and operators query a single events table. Higher effort, higher payoff.

Recommended: A1 in v1, A2 if v1 friction is high.

### B. Operator dashboard (CloudWatch Dashboard via CDK)

A purpose-built CloudWatch Dashboard surfacing pipeline-level metrics:

- **Pipeline success rate** — % of `agent:pr-completed` vs `agent:error` per day / week.
- **Time per stage** — explore / clarification / implement / critique / pr stage durations, p50/p90/p99.
- **Cost per issue** — sum of `claude_code.api_request.cost_usd` per session ID.
- **Re-invocation count distribution** — histogram of how many invocations per issue.
- **microVM lifecycle events** — count of `stop_runtime_session` calls (post-PR-#32), maxLifetime reaps inferred from microVM uptime resets.
- **Recent failed runs** — table of last N Step Functions executions with status FAILED.
- **Top errors** — log-insights aggregation of error patterns (HTTP 4xx/5xx from AgentCore, `traceback` lines in Lambda logs, etc.).

CDK construct in `lib/constructs/observability/operator-dashboard.ts`. New nested stack `observability-stack.ts`. CfnOutput the dashboard URL.

### C. Alerting (CloudWatch Alarms + SNS)

Operator-actionable alarms:

- **Step Functions failures** — alarm on FAILED executions; threshold = 1 in 5min.
- **Lambda errors** — alarm on Setup or Pipeline Lambda error rate > 0 in 5min.
- **microVM HTTP 4xx/5xx from `commands` API** — alarm on the 400 family (catches issues like the one PR #30 fixed before users notice).
- **agent:error label rate** — alarm if N+ issues land with `agent:error` in 1h (signals systemic failure mode).
- **Long-running runs** — alarm if a Step Functions execution exceeds 30 min (anything that's about to maxLifetime-reap).
- **Stale-base local branches** — once #34 lands, alarm on the new error type it surfaces.

SNS topic `sdlc-pipeline-operator-alerts` configurable in `sdlc-config.yaml` to subscribe email / Slack / PagerDuty. CDK construct in `lib/constructs/observability/alarms.ts`.

## Decomposition

| # | Title | Scope | Estimated complexity | Dependencies |
|---|---|---|---|---|
| C1 | feat: pipeline-run trace via CloudWatch Logs Insights saved queries (component A1) | CDK only — saved queries resourced as `cloudwatch.QueryDefinition`. ~3 files. | Path A | None. Independent. |
| C2 | feat: operator CloudWatch Dashboard for SDLC pipeline (component B) | New nested stack + dashboard construct + 7 widgets. CDK only. | Path B | None. Can parallel with C1. |
| C3 | feat: SNS alarms for SDLC pipeline error states (component C) | New construct + SNS topic + 6 alarms. CDK + config schema field for subscriber. | Path B | None. |
| C4 | feat: pipeline-trace Lambda for cross-source timeline (component A2) | If A1 friction is high after operating C1 for a while. New Lambda + IAM + CDK + Python. | Path B | C1 should be tried first. |

## Constraints applying to ALL child issues

- All Python files keep the Apache-2.0 header.
- CDK: cdk-nag passes on every new construct. Suppressions only with inline justification.
- New env vars / config fields go into `sdlc-config.template.yaml` with documentation.
- Don't introduce new third-party Python or Node dependencies for the alarm/dashboard work — `aws-cdk-lib` and existing constructs are sufficient.
- Don't modify existing Lambda code paths to add observability hooks. The existing `print` lines + claude OTel events are the data source; observability work *consumes* them, doesn't generate new ones.
- Each child issue uses Path B if multi-file or architectural; Path A if a single CDK construct addition.

## Verification protocol applied to every child issue

- [ ] `npx cdk synth --quiet` passes; quote relevant template snippets in PR body.
- [ ] For dashboard / alarm work: live verification against the deployed account — list the dashboard widgets and the alarm states post-deploy.
- [ ] cdk-nag passes; any suppression has inline justification.
- [ ] `## Known Interactions` section in PR body explains how the new resource interacts with existing ones (e.g., dashboard reads from the same log groups the Setup Lambda writes to; alarm thresholds chosen relative to current pipeline run cadence).

## Reference material

- `coding-assistants/claude-code/obs.md` — full claude OTel architecture, attribute schema, useful Logs Insights queries. The dashboard widgets in C2 should be informed by this.
- `lib/nested/assistant-stack.ts` — Setup + Pipeline Lambda definitions; their log groups are the data source for the trace and dashboard.
- AWS docs: [AgentCore Observability](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-configure.html), [CloudWatch Dashboards CDK](https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws_cloudwatch.Dashboard.html).
- This session's `.claude/skills/test-runtime-coding-agent/SKILL.md` — encodes manual probe patterns we should make redundant via dashboards.

## Decision the user makes after reading this

1. Approve the three-component architecture (A / B / C) or revise.
2. Pick which child issues to file first. **Suggested order: C3 (alarms) → C2 (dashboard) → C1 (saved queries) → C4 (trace Lambda, only if needed).** Alarms have the highest delta-utility per LOC; dashboards are the next-highest; saved queries are convenience; trace Lambda is the most code and only worth it if the simpler tools fall short.
3. Tell me to file the child issues — at which point I file them via the create-sdlc-issue skill, individually, with the appropriate `agent:start` cadence.

This issue stays open as the parent / tracking issue. When all child issues are merged, close this one.

Start with coding-assistants/claude-code/obs.md and lib/nested/assistant-stack.ts to understand the existing observability data sources and Lambda log groups. Review the C1-C4 decomposition and the stated constraints; this issue is done when the architecture is approved or revised and the selected child implementation issues are filed.

Written by the indexing model from the issue text.