aws-samples / aws-samples/sample-autonomous-cloud-coding-agents

CDK: Harness-level SLO metrics on operator dashboard

Abierto
#511 0 comentarios 0 reacciones 0 asignados Ver en GitHub
enhancement infra-cdk P1
Lenguaje dominante
TypeScript
Estrellas
143
Forks
46
Merge medio
3 d 9 h
PR fusionados (30 d)
20

Descripción

> **Roadmap:** Evaluation pipeline; Validation and risk analytics
> **Priority:** P1

## Component

CDK / infrastructure

## Describe the feature

Emit **harness-level** CloudWatch metrics and dashboard panels that evaluate the control plane/runtime separately from raw task pass/fail. Final task success conflates model quality, harness quality, and environment luck (arXiv:2605.18747 §5.2.1).

## Use case

- **Operators/SREs** track whether the platform is getting more reliable independent of model upgrades.
- **Field engagements** report dark-factory scorecard progress with objective harness SLOs.
- **A/B prompt experiments** isolate prompt changes from harness regressions.

## Proposed solution

Define and emit metrics (names illustrative; finalize in implementation):

| Metric | Definition |
|--------|------------|
| `HarnessToolCallsPerTask` | Agent tool invocations per completed task |
| `HarnessTokensPerTask` | Bedrock tokens (from existing cost telemetry) |
| `HarnessWallClockSeconds` | Submit → terminal |
| `HarnessVerificationCoverage` | % tasks with full evidence bundle / all required sensors run |
| `HarnessRecoveryRate` | % tasks that failed verify then succeeded on retry (when fix-up loop exists) |
| `HarnessReplayCompleteness` | % tasks with TaskEvents + trace URI + prompt_version |

Add widgets to the existing operator dashboard construct; document dimensions (`repo`, `workflow_ref`, `prompt_version`) without PII.

### Acceptance criteria

- [ ] Metrics emitted from orchestrator and/or agent telemetry path
- [ ] Dashboard panels added to operator dashboard (CDK)
- [ ] Documented in `docs/design/OBSERVABILITY.md` with metric definitions
- [ ] Referenced from dark-factory scorecard / `ABCA_V2.md` success metrics where appropriate
- [ ] No new PII dimensions in metric labels

## Other information

- **Depends on (soft):** Verification evidence bundles for `VerificationCoverage`
- **Paper:** arXiv:2605.18747 §5.2.1 harness-level evaluation dimensions
- **Existing:** Operator dashboard, OTEL spans, TaskEvents (shipped)

## Acknowledgements

- [ ] I may be able to implement this feature
- [ ] This might be a breaking change

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comienza con el constructo existente del dashboard del operador y la ruta de telemetría del orquestador o agente; después, revisa los spans de OTEL existentes, TaskEvents y la telemetría de costes. Define las métricas del harness y las dimensiones no relacionadas con PII, añade paneles al dashboard, documéntalos en docs/design/OBSERVABILITY.md y actualiza, cuando corresponda, las métricas de éxito del scorecard de dark-factory o de ABCA_V2.md.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
aws, typescript
Área
cloud, infrastructure, observability-sre
Tipo de issue
Nueva funcionalidad
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
48/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.