AltimateAI / AltimateAI/altimate-code

agent_outcome telemetry has empty reason/final_tool/error_class for ~30% of builder runs

Abierto
#765 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
TypeScript
Estrellas
811
Forks
134
Merge medio
3 d 2 h
PR fusionados (30 d)
50

Descripción

## Problem

Telemetry analysis (2026-04-28, last 7 PDT days) shows the `agent_outcome` event ships with `reason`, `final_tool`, and `error_class` all empty. About 30% of builder runs end with `outcome` ∈ {`abandoned`, `aborted`, `error`}, so today ~30% of builder failures are undiagnosable from telemetry alone.

Concretely, builder completion rate has been stuck at 62-69% week-over-week, but we can't tell *why* failures happen without falling back to per-session trace inspection.

## Evidence

Querying Azure App Insights workspace `b511e30e-4b93-4093-98a5-b80fc4718111`:

```kql
AppEvents
| where TimeGenerated >= ago(7d) and Name == 'agent_outcome'
| extend agent=tostring(Properties.agent),
outcome=tostring(Properties.outcome),
reason=tostring(Properties.reason),
final_tool=tostring(Properties.final_tool),
error_class=tostring(Properties.error_class)
| where agent == 'builder' and outcome != 'completed'
| summarize cnt=count() by outcome, reason, final_tool, error_class
```

Returns 477 builder failures, all with `reason=""`, `final_tool=""`, `error_class=""`.

## Fix

Wire diagnostic context end-to-end through a pure helper:
- Track `lastToolName` alongside existing `lastToolCategory` in `prompt.ts`
- Capture `error.data.message` (only) at the error site, with `maskString` applied at extraction
- Pass `errorRecords[].errorClass` through for `aborted` outcomes
- Refine `abort.reason` extraction to handle Error / object / string variants safely
- Apply `maskString` symmetrically to both `error` and `aborted` reasons

## Telemetry contract

Post-fix, `agent_outcome` events will include three required string fields:
- `final_tool`: name of last tool invoked (empty if none)
- `error_class`: classification (`file_not_found`, `connection`, `timeout`, etc.) or empty
- `reason`: short masked diagnostic snippet, or `"no_tools_invoked"` / `"user_cancelled"` / `""`

## Privacy

`error.data` for variants like `APIError` carries `responseBody`, `responseHeaders`, `metadata` — provider auth tokens, raw response bodies, request URLs. The fix extracts only `error.data.message` (human-readable) and runs `maskString` at the extraction site to prevent any leak vector.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.