AltimateAI / AltimateAI/altimate-code
agent_outcome telemetry has empty reason/final_tool/error_class for ~30% of builder runs
- Langage dominant
- TypeScript
- Étoiles
- 811
- Forks
- 134
- Merge moyen
- 3 j 2 h
- PR mergées (30 j)
- 50
Description
## Problem
Telemetry analysis (2026-04-28, last 7 PDT days) shows the `agent_outcome` event ships with `reason`, `final_tool`, and `error_class` all empty. About 30% of builder runs end with `outcome` ∈ {`abandoned`, `aborted`, `error`}, so today ~30% of builder failures are undiagnosable from telemetry alone.
Concretely, builder completion rate has been stuck at 62-69% week-over-week, but we can't tell *why* failures happen without falling back to per-session trace inspection.
## Evidence
Querying Azure App Insights workspace `b511e30e-4b93-4093-98a5-b80fc4718111`:
```kql
AppEvents
| where TimeGenerated >= ago(7d) and Name == 'agent_outcome'
| extend agent=tostring(Properties.agent),
outcome=tostring(Properties.outcome),
reason=tostring(Properties.reason),
final_tool=tostring(Properties.final_tool),
error_class=tostring(Properties.error_class)
| where agent == 'builder' and outcome != 'completed'
| summarize cnt=count() by outcome, reason, final_tool, error_class
```
Returns 477 builder failures, all with `reason=""`, `final_tool=""`, `error_class=""`.
## Fix
Wire diagnostic context end-to-end through a pure helper:
- Track `lastToolName` alongside existing `lastToolCategory` in `prompt.ts`
- Capture `error.data.message` (only) at the error site, with `maskString` applied at extraction
- Pass `errorRecords[].errorClass` through for `aborted` outcomes
- Refine `abort.reason` extraction to handle Error / object / string variants safely
- Apply `maskString` symmetrically to both `error` and `aborted` reasons
## Telemetry contract
Post-fix, `agent_outcome` events will include three required string fields:
- `final_tool`: name of last tool invoked (empty if none)
- `error_class`: classification (`file_not_found`, `connection`, `timeout`, etc.) or empty
- `reason`: short masked diagnostic snippet, or `"no_tools_invoked"` / `"user_cancelled"` / `""`
## Privacy
`error.data` for variants like `APIError` carries `responseBody`, `responseHeaders`, `metadata` — provider auth tokens, raw response bodies, request URLs. The fix extracts only `error.data.message` (human-readable) and runs `maskString` at the extraction site to prevent any leak vector.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.