Token cost as first-class data: per-doc in graph.json, cumulative on trails, total for llms.txt
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Why
Agents budget tokens, not clicks. Every navigational artifact matlatl emits invites an agent to read documents, but nothing states what that reading costs. A trail that says "9 docs, ~14k tokens" lets a consumer decide rationally whether to follow it; a bare list doesn't. This also directly mitigates the over-reading failure mode the ETH AGENTS.md studies measured (arXiv:2601.20404).
What to build
A deterministic token estimate (pinned approximation — e.g. ceil(bytes/4) or a fixed word/punctuation rule; document it as an estimate, never a tokenizer dependency):
- graph.json: per-node
approxTokens(int — no float concerns). - trails.json: per-entry cumulative and per-trail total, so consumers can truncate a trail at budget.
- llms.txt: total corpus estimate in the summary blockquote; optionally per-entry annotations.
- Human report: corpus total in the summary line.
Schema bumps: graph.json v6→v7, trails.json v1→v2 (additive).
Constraints
- Integer arithmetic only — byte-stable trivially, no Float wire type needed.
- Domain purity (ADR 0004): the estimate is a pure function over the already-loaded document bytes/text.
- Do NOT add a real tokenizer dependency; the value is ballpark budgeting, and a pinned approximation keeps determinism and zero deps.
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the generators for graph.json, trails.json, llms.txt, and the human report, then read ADR 0004 and trace where document bytes or text are already loaded. Define and document the pinned integer approximation there, and verify that graph.json v7, trails.json v2, the llms.txt summary, and the human report expose the requested estimates deterministically.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100