aws-samples / aws-samples/sample-autonomous-cloud-coding-agents
CDK: Harness-level SLO metrics on operator dashboard
- Lenguaje dominante
- TypeScript
- Estrellas
- 143
- Forks
- 46
- Merge medio
- 3 d 9 h
- PR fusionados (30 d)
- 20
Descripción
> **Roadmap:** Evaluation pipeline; Validation and risk analytics
> **Priority:** P1
## Component
CDK / infrastructure
## Describe the feature
Emit **harness-level** CloudWatch metrics and dashboard panels that evaluate the control plane/runtime separately from raw task pass/fail. Final task success conflates model quality, harness quality, and environment luck (arXiv:2605.18747 §5.2.1).
## Use case
- **Operators/SREs** track whether the platform is getting more reliable independent of model upgrades.
- **Field engagements** report dark-factory scorecard progress with objective harness SLOs.
- **A/B prompt experiments** isolate prompt changes from harness regressions.
## Proposed solution
Define and emit metrics (names illustrative; finalize in implementation):
| Metric | Definition |
|--------|------------|
| `HarnessToolCallsPerTask` | Agent tool invocations per completed task |
| `HarnessTokensPerTask` | Bedrock tokens (from existing cost telemetry) |
| `HarnessWallClockSeconds` | Submit → terminal |
| `HarnessVerificationCoverage` | % tasks with full evidence bundle / all required sensors run |
| `HarnessRecoveryRate` | % tasks that failed verify then succeeded on retry (when fix-up loop exists) |
| `HarnessReplayCompleteness` | % tasks with TaskEvents + trace URI + prompt_version |
Add widgets to the existing operator dashboard construct; document dimensions (`repo`, `workflow_ref`, `prompt_version`) without PII.
### Acceptance criteria
- [ ] Metrics emitted from orchestrator and/or agent telemetry path
- [ ] Dashboard panels added to operator dashboard (CDK)
- [ ] Documented in `docs/design/OBSERVABILITY.md` with metric definitions
- [ ] Referenced from dark-factory scorecard / `ABCA_V2.md` success metrics where appropriate
- [ ] No new PII dimensions in metric labels
## Other information
- **Depends on (soft):** Verification evidence bundles for `VerificationCoverage`
- **Paper:** arXiv:2605.18747 §5.2.1 harness-level evaluation dimensions
- **Existing:** Operator dashboard, OTEL spans, TaskEvents (shipped)
## Acknowledgements
- [ ] I may be able to implement this feature
- [ ] This might be a breaking change
Guía de contribución
Línea de trabajo
Comienza con el constructo existente del dashboard del operador y la ruta de telemetría del orquestador o agente; después, revisa los spans de OTEL existentes, TaskEvents y la telemetría de costes. Define las métricas del harness y las dimensiones no relacionadas con PII, añade paneles al dashboard, documéntalos en docs/design/OBSERVABILITY.md y actualiza, cuando corresponda, las métricas de éxito del scorecard de dark-factory o de ABCA_V2.md.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- aws, typescript
- Área
- cloud, infrastructure, observability-sre
- Tipo de issue
- Nueva funcionalidad
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Tranquilo
- Claridad
- Bastante claro
- Aptitud para principiantes
- 48/100