aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
CDK: Harness-level SLO metrics on operator dashboard
- Lingua principale
- TypeScript
- Stelle
- 143
- Fork
- 46
- Merge medio
- 3g 9h
- PR unite (30g)
- 20
Descrizione
> **Roadmap:** Evaluation pipeline; Validation and risk analytics
> **Priority:** P1
## Component
CDK / infrastructure
## Describe the feature
Emit **harness-level** CloudWatch metrics and dashboard panels that evaluate the control plane/runtime separately from raw task pass/fail. Final task success conflates model quality, harness quality, and environment luck (arXiv:2605.18747 §5.2.1).
## Use case
- **Operators/SREs** track whether the platform is getting more reliable independent of model upgrades.
- **Field engagements** report dark-factory scorecard progress with objective harness SLOs.
- **A/B prompt experiments** isolate prompt changes from harness regressions.
## Proposed solution
Define and emit metrics (names illustrative; finalize in implementation):
| Metric | Definition |
|--------|------------|
| `HarnessToolCallsPerTask` | Agent tool invocations per completed task |
| `HarnessTokensPerTask` | Bedrock tokens (from existing cost telemetry) |
| `HarnessWallClockSeconds` | Submit → terminal |
| `HarnessVerificationCoverage` | % tasks with full evidence bundle / all required sensors run |
| `HarnessRecoveryRate` | % tasks that failed verify then succeeded on retry (when fix-up loop exists) |
| `HarnessReplayCompleteness` | % tasks with TaskEvents + trace URI + prompt_version |
Add widgets to the existing operator dashboard construct; document dimensions (`repo`, `workflow_ref`, `prompt_version`) without PII.
### Acceptance criteria
- [ ] Metrics emitted from orchestrator and/or agent telemetry path
- [ ] Dashboard panels added to operator dashboard (CDK)
- [ ] Documented in `docs/design/OBSERVABILITY.md` with metric definitions
- [ ] Referenced from dark-factory scorecard / `ABCA_V2.md` success metrics where appropriate
- [ ] No new PII dimensions in metric labels
## Other information
- **Depends on (soft):** Verification evidence bundles for `VerificationCoverage`
- **Paper:** arXiv:2605.18747 §5.2.1 harness-level evaluation dimensions
- **Existing:** Operator dashboard, OTEL spans, TaskEvents (shipped)
## Acknowledgements
- [ ] I may be able to implement this feature
- [ ] This might be a breaking change
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia con il costrutto esistente della dashboard dell’operatore e il percorso di telemetria dell’orchestrator o dell’agent, quindi esamina gli span OTEL esistenti, i TaskEvents e la telemetria dei costi. Definisci le metriche dell’harness e le dimensioni non PII, aggiungi pannelli alla dashboard, documentali in docs/design/OBSERVABILITY.md e aggiorna, ove appropriato, le metriche di successo della scorecard dark-factory o di ABCA_V2.md.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- aws, typescript
- Ambito
- cloud, infrastructure, observability-sre
- Tipo di issue
- Funzionalità
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Tranquilla
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 48/100