jobizzness / jobizzness/bench

Devin turns report zero tokens and zero cost — the usage data is on the ACP wire and is being dropped

Open
#115 8 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
1
Forks
0
Avg merge
1h 21m
Merged PRs (30d)
41

Description

## What is wrong

Every Devin turn reports zero tokens, zero cost and no context usage. The code says that data
does not exist. It does — the shipped binary puts it in the `session/prompt` result.

### The evidence

`src/daemon/devin-session.ts`:

```ts
// ACP exposes no documented per-turn token feed for Devin.
get turnTokens(): number { return 0; } // :112-113

// Devin exposes no documented context-window figure Bench can consume.
get contextUsed(): Context | null { return null; } // :117-118

// Devin bills in ACUs and ACP has no documented per-turn dollar cost.
// :373 — no total_cost_usd on the result
```

Driving the real `devin acp` on this machine with the same handshake `DevinSession` uses, the
result of `session/prompt` was:

```json
{"stopReason": "end_turn",
"usage": {"totalTokens": 10952, "inputTokens": 10920, "outputTokens": 32,
"cachedReadTokens": 8192},
"_meta": {"cognition.ai/userMessageId": "e7639c85-9c33-4f53-9771-dad6528565ba"}}
```

Also observed on the wire during that turn, before the result:

- two `session/update` notifications with `sessionUpdate: "usage_update"`
- a `_cognition.ai/turn_stats` notification

And in the binary's method/field table: `cognition.ai/acuUsed`, `cognition.ai/acu`,
`cognition.ai/billingInformation`.

`DevinSession.handle()` (`src/daemon/devin-session.ts:206-262`) ignores all of it — the result
branch reads only `stopReason`, and `update()` (`:290-307`) handles only `agent_message_chunk`
and the two tool-call kinds.

The three comments quoted above were written from documentation research
(`.bench/reports/d2471cb8-.../1/devin-findings.md` §7, which correctly said no *documented*
per-turn feed exists). They are true of the docs and false of the binary.

## Acceptance criteria

- [ ] `turnTokens` reports the real figure for the running turn, fed from `usage_update` so it
moves during the turn rather than only at the end — that is what the roster's live token
count is for.
- [ ] The `usage` block on the `session/prompt` result is read and carried onto the `ResultEvent`
so the turn is recorded with its true token counts.
- [ ] `contextUsed` reports something real if the wire gives enough to compute it, or stays
`null` with a comment saying what is actually missing. Do not leave a comment that says the
data does not exist when it does.
- [ ] If an ACU or dollar figure is on the wire (`cognition.ai/acuUsed`,
`cognition.ai/turn_stats`, `cognition.ai/billingInformation`), Devin turns are priced with
it. If after looking it is not there, say so on this issue with what you saw — do not
silently leave the cost at zero.
- [ ] Unknown/absent usage stays absent rather than becoming a confident zero. A Devin tab that
reports `$0.00` when it has actually spent is worse than one that reports nothing.
- [ ] Tests drive a fake ACP server that emits `usage_update` and a result carrying `usage`.

## Out of scope

- The Devin model picker (#114) and the resume crash (#113).
- Any change to how Claude turns are priced, or to the shared ledger/spend format beyond what
carrying a second runtime's numbers requires.
- Cloud/org-level Devin billing APIs. This is about the per-turn feed already on the stdio wire.

## Verification

```
pnpm test
pnpm exec tsc -p tsconfig.json --noEmit
```

Manual check a green build will not catch: run one real Devin turn from the cockpit and confirm
the roster's token count moves during the turn and the finished turn is recorded with non-zero
tokens.

## Related

- #113, #114 — same file, keep the diffs apart.
- Prior research: `.bench/reports/d2471cb8-45f1-4b17-82bf-08c8d471c803/1/devin-findings.md` §7.

Contributor guide

Open the contributing guide

Research direction

Start in src/daemon/devin-session.ts, especially handle() and update(), then inspect the fake ACP server used by the tests. Trace usage_update notifications and the session/prompt result into the live turn and ResultEvent data, including how context and pricing are represented. Run pnpm test and pnpm exec tsc -p tsconfig.json --noEmit; done means tests cover usage and real Devin turns no longer report known usage as zero.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend-api-design, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.