aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

CDK: Harness-level SLO metrics on operator dashboard

Open
#511 0 comments 0 reactions 0 assignees View on GitHub
enhancement infra-cdk P1
Dominant language
TypeScript
Stars
143
Forks
46
Avg merge
3d 9h
Merged PRs (30d)
20

Description

> **Roadmap:** Evaluation pipeline; Validation and risk analytics
> **Priority:** P1

## Component

CDK / infrastructure

## Describe the feature

Emit **harness-level** CloudWatch metrics and dashboard panels that evaluate the control plane/runtime separately from raw task pass/fail. Final task success conflates model quality, harness quality, and environment luck (arXiv:2605.18747 §5.2.1).

## Use case

- **Operators/SREs** track whether the platform is getting more reliable independent of model upgrades.
- **Field engagements** report dark-factory scorecard progress with objective harness SLOs.
- **A/B prompt experiments** isolate prompt changes from harness regressions.

## Proposed solution

Define and emit metrics (names illustrative; finalize in implementation):

| Metric | Definition |
|--------|------------|
| `HarnessToolCallsPerTask` | Agent tool invocations per completed task |
| `HarnessTokensPerTask` | Bedrock tokens (from existing cost telemetry) |
| `HarnessWallClockSeconds` | Submit → terminal |
| `HarnessVerificationCoverage` | % tasks with full evidence bundle / all required sensors run |
| `HarnessRecoveryRate` | % tasks that failed verify then succeeded on retry (when fix-up loop exists) |
| `HarnessReplayCompleteness` | % tasks with TaskEvents + trace URI + prompt_version |

Add widgets to the existing operator dashboard construct; document dimensions (`repo`, `workflow_ref`, `prompt_version`) without PII.

### Acceptance criteria

- [ ] Metrics emitted from orchestrator and/or agent telemetry path
- [ ] Dashboard panels added to operator dashboard (CDK)
- [ ] Documented in `docs/design/OBSERVABILITY.md` with metric definitions
- [ ] Referenced from dark-factory scorecard / `ABCA_V2.md` success metrics where appropriate
- [ ] No new PII dimensions in metric labels

## Other information

- **Depends on (soft):** Verification evidence bundles for `VerificationCoverage`
- **Paper:** arXiv:2605.18747 §5.2.1 harness-level evaluation dimensions
- **Existing:** Operator dashboard, OTEL spans, TaskEvents (shipped)

## Acknowledgements

- [ ] I may be able to implement this feature
- [ ] This might be a breaking change

Contributor guide

Open the contributing guide

Research direction

Start with the existing operator dashboard construct and the orchestrator or agent telemetry path, then review existing OTEL spans, TaskEvents, and cost telemetry. Define the harness metrics and non-PII dimensions, add dashboard panels, document them in docs/design/OBSERVABILITY.md, and update dark-factory scorecard or ABCA_V2.md success metrics where appropriate.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.