HarperFast / HarperFast/harper

Add a loop-level summary (iterations / tokens / cost / terminal reason) to toolMode:'auto' results

Open
#850 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Context

The `toolMode: 'auto'` agent loop (#612, PR HarperFast/harper#848) records a per-tool-invocation `trace: ToolTraceEntry[]` — one entry per tool call, with arguments, result, truncation, duration, and per-call errors. That's the right granularity for "which tools ran and what did each return."

What it does **not** yet capture is a loop-level summary: the answers to "how many iterations did this take, how much did it cost, and why did it stop." Today those are reconstructable only by:
- counting `trace` entries (≈ iterations, but not exact — a terminal round with no tool call leaves no entry), and
- cross-querying `hdb_model_calls` rows by request/conversation for token totals.

For a primitive meant to be transparent and debuggable — and one that HarperFast/harper-pro#676's built-in Agent will lean on for per-session budgets — a single structured summary on the result is worth having.

## Proposal

Add an optional loop-summary object to `GenerateResult` (auto mode only), populated alongside `trace`:

```ts
interface ToolLoopSummary {
iterations: number; // backend rounds actually run
toolCallCount: number; // total tool calls dispatched
totalTokens?: number; // cumulative prompt+completion (when backend reports usage)
totalCostUsd?: number; // cumulative cost (when a rate card is wired)
terminalReason: 'stop' | 'length' | 'content_filter'
| 'iterations' | 'tokens' | 'cost' | 'abort' | 'tool_error';
budgetUnmeasurable?: boolean; // a budget was set but usage wasn't reported
}
```

- Populated on the success path when `includeToolTrace` is set (same gate as `trace`).
- **Always** attached on the error/abort paths (mirror the existing `partialTrace` posture on `BudgetExceededError` / `ToolHandlerError`) — operators need the terminal reason precisely when something went wrong.
- `terminalReason` unifies the model's `finishReason` with the loop's own stop conditions (budget kind, abort, tool-error abort) into one field, so "why did it stop" has a single authoritative answer.

## Why separate from this PR

Deliberately deferred out of the HarperFast/harper#848 hardening pass: it's a `GenerateResult` surface addition (and a corresponding shape for the error classes), and it's worth landing on its own rather than slipping into a bug-fix commit. The hardening pass already wired the underlying signals (cumulative token/cost totals, budget-unmeasurable warn-once, per-iteration counts) — this issue is mostly about surfacing them in a stable shape.

## Acceptance

- [ ] `ToolLoopSummary` (or equivalent) defined on the type surface, auto-mode only.
- [ ] Populated on success when `includeToolTrace` is set; always on error/abort paths.
- [ ] `terminalReason` covers both model finish reasons and loop stop conditions.
- [ ] `budgetUnmeasurable` set when a budget was configured but backend usage was absent.
- [ ] Streaming path: decide whether the summary is exposed (the stream returns no result object today — likely a terminal metadata chunk or out of scope for v1).
- [ ] Tests across terminal-first, multi-round, each budget trip, abort, and tool-error-abort.

## Related

- Follows HarperFast/harper#612 / PR HarperFast/harper#848 (the agent loop + its resilience hardening).
- Downstream consumer: HarperFast/harper-pro#676 (built-in Agent) — per-session budget reporting reads naturally off this.

---

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.