Devin turns report zero tokens and zero cost — the usage data is on the ACP wire and is being dropped
- Dominant language
- TypeScript
- Stars
- 1
- Forks
- 0
- Avg merge
- 1h 21m
- Merged PRs (30d)
- 41
Description
## What is wrong
Every Devin turn reports zero tokens, zero cost and no context usage. The code says that data
does not exist. It does — the shipped binary puts it in the `session/prompt` result.
### The evidence
`src/daemon/devin-session.ts`:
```ts
// ACP exposes no documented per-turn token feed for Devin.
get turnTokens(): number { return 0; } // :112-113
// Devin exposes no documented context-window figure Bench can consume.
get contextUsed(): Context | null { return null; } // :117-118
// Devin bills in ACUs and ACP has no documented per-turn dollar cost.
// :373 — no total_cost_usd on the result
```
Driving the real `devin acp` on this machine with the same handshake `DevinSession` uses, the
result of `session/prompt` was:
```json
{"stopReason": "end_turn",
"usage": {"totalTokens": 10952, "inputTokens": 10920, "outputTokens": 32,
"cachedReadTokens": 8192},
"_meta": {"cognition.ai/userMessageId": "e7639c85-9c33-4f53-9771-dad6528565ba"}}
```
Also observed on the wire during that turn, before the result:
- two `session/update` notifications with `sessionUpdate: "usage_update"`
- a `_cognition.ai/turn_stats` notification
And in the binary's method/field table: `cognition.ai/acuUsed`, `cognition.ai/acu`,
`cognition.ai/billingInformation`.
`DevinSession.handle()` (`src/daemon/devin-session.ts:206-262`) ignores all of it — the result
branch reads only `stopReason`, and `update()` (`:290-307`) handles only `agent_message_chunk`
and the two tool-call kinds.
The three comments quoted above were written from documentation research
(`.bench/reports/d2471cb8-.../1/devin-findings.md` §7, which correctly said no *documented*
per-turn feed exists). They are true of the docs and false of the binary.
## Acceptance criteria
- [ ] `turnTokens` reports the real figure for the running turn, fed from `usage_update` so it
moves during the turn rather than only at the end — that is what the roster's live token
count is for.
- [ ] The `usage` block on the `session/prompt` result is read and carried onto the `ResultEvent`
so the turn is recorded with its true token counts.
- [ ] `contextUsed` reports something real if the wire gives enough to compute it, or stays
`null` with a comment saying what is actually missing. Do not leave a comment that says the
data does not exist when it does.
- [ ] If an ACU or dollar figure is on the wire (`cognition.ai/acuUsed`,
`cognition.ai/turn_stats`, `cognition.ai/billingInformation`), Devin turns are priced with
it. If after looking it is not there, say so on this issue with what you saw — do not
silently leave the cost at zero.
- [ ] Unknown/absent usage stays absent rather than becoming a confident zero. A Devin tab that
reports `$0.00` when it has actually spent is worse than one that reports nothing.
- [ ] Tests drive a fake ACP server that emits `usage_update` and a result carrying `usage`.
## Out of scope
- The Devin model picker (#114) and the resume crash (#113).
- Any change to how Claude turns are priced, or to the shared ledger/spend format beyond what
carrying a second runtime's numbers requires.
- Cloud/org-level Devin billing APIs. This is about the per-turn feed already on the stdio wire.
## Verification
```
pnpm test
pnpm exec tsc -p tsconfig.json --noEmit
```
Manual check a green build will not catch: run one real Devin turn from the cockpit and confirm
the roster's token count moves during the turn and the finished turn is recorded with non-zero
tokens.
## Related
- #113, #114 — same file, keep the diffs apart.
- Prior research: `.bench/reports/d2471cb8-45f1-4b17-82bf-08c8d471c803/1/devin-findings.md` §7.
Contributor guide
Research direction
Start in src/daemon/devin-session.ts, especially handle() and update(), then inspect the fake ACP server used by the tests. Trace usage_update notifications and the session/prompt result into the live turn and ResultEvent data, including how context and pricing are represented. Run pnpm test and pnpm exec tsc -p tsconfig.json --noEmit; done means tests cover usage and real Devin turns no longer report known usage as zero.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend-api-design, devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100