aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

CDK: Harness-level SLO metrics on operator dashboard

Aperta
#511 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
enhancement infra-cdk P1
Lingua principale
TypeScript
Stelle
143
Fork
46
Merge medio
3g 9h
PR unite (30g)
20

Descrizione

> **Roadmap:** Evaluation pipeline; Validation and risk analytics
> **Priority:** P1

## Component

CDK / infrastructure

## Describe the feature

Emit **harness-level** CloudWatch metrics and dashboard panels that evaluate the control plane/runtime separately from raw task pass/fail. Final task success conflates model quality, harness quality, and environment luck (arXiv:2605.18747 §5.2.1).

## Use case

- **Operators/SREs** track whether the platform is getting more reliable independent of model upgrades.
- **Field engagements** report dark-factory scorecard progress with objective harness SLOs.
- **A/B prompt experiments** isolate prompt changes from harness regressions.

## Proposed solution

Define and emit metrics (names illustrative; finalize in implementation):

| Metric | Definition |
|--------|------------|
| `HarnessToolCallsPerTask` | Agent tool invocations per completed task |
| `HarnessTokensPerTask` | Bedrock tokens (from existing cost telemetry) |
| `HarnessWallClockSeconds` | Submit → terminal |
| `HarnessVerificationCoverage` | % tasks with full evidence bundle / all required sensors run |
| `HarnessRecoveryRate` | % tasks that failed verify then succeeded on retry (when fix-up loop exists) |
| `HarnessReplayCompleteness` | % tasks with TaskEvents + trace URI + prompt_version |

Add widgets to the existing operator dashboard construct; document dimensions (`repo`, `workflow_ref`, `prompt_version`) without PII.

### Acceptance criteria

- [ ] Metrics emitted from orchestrator and/or agent telemetry path
- [ ] Dashboard panels added to operator dashboard (CDK)
- [ ] Documented in `docs/design/OBSERVABILITY.md` with metric definitions
- [ ] Referenced from dark-factory scorecard / `ABCA_V2.md` success metrics where appropriate
- [ ] No new PII dimensions in metric labels

## Other information

- **Depends on (soft):** Verification evidence bundles for `VerificationCoverage`
- **Paper:** arXiv:2605.18747 §5.2.1 harness-level evaluation dimensions
- **Existing:** Operator dashboard, OTEL spans, TaskEvents (shipped)

## Acknowledgements

- [ ] I may be able to implement this feature
- [ ] This might be a breaking change

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia con il costrutto esistente della dashboard dell’operatore e il percorso di telemetria dell’orchestrator o dell’agent, quindi esamina gli span OTEL esistenti, i TaskEvents e la telemetria dei costi. Definisci le metriche dell’harness e le dimensioni non PII, aggiungi pannelli alla dashboard, documentali in docs/design/OBSERVABILITY.md e aggiorna, ove appropriato, le metriche di successo della scorecard dark-factory o di ABCA_V2.md.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, typescript
Ambito
cloud, infrastructure, observability-sre
Tipo di issue
Funzionalità
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Tranquilla
Chiarezza
Abbastanza chiara
Idoneità per principianti
48/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.