stacklok / stacklok/matlatl

Token cost as first-class data: per-doc in graph.json, cumulative on trails, total for llms.txt

Open
#21 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Why

Agents budget tokens, not clicks. Every navigational artifact matlatl emits invites an agent to read documents, but nothing states what that reading costs. A trail that says "9 docs, ~14k tokens" lets a consumer decide rationally whether to follow it; a bare list doesn't. This also directly mitigates the over-reading failure mode the ETH AGENTS.md studies measured (arXiv:2601.20404).

What to build

A deterministic token estimate (pinned approximation — e.g. ceil(bytes/4) or a fixed word/punctuation rule; document it as an estimate, never a tokenizer dependency):

  1. graph.json: per-node approxTokens (int — no float concerns).
  2. trails.json: per-entry cumulative and per-trail total, so consumers can truncate a trail at budget.
  3. llms.txt: total corpus estimate in the summary blockquote; optionally per-entry annotations.
  4. Human report: corpus total in the summary line.

Schema bumps: graph.json v6→v7, trails.json v1→v2 (additive).

Constraints

  • Integer arithmetic only — byte-stable trivially, no Float wire type needed.
  • Domain purity (ADR 0004): the estimate is a pure function over the already-loaded document bytes/text.
  • Do NOT add a real tokenizer dependency; the value is ballpark budgeting, and a pinned approximation keeps determinism and zero deps.

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the generators for graph.json, trails.json, llms.txt, and the human report, then read ADR 0004 and trace where document bytes or text are already loaded. Define and document the pinned integer approximation there, and verify that graph.json v7, trails.json v2, the llms.txt summary, and the human report expose the requested estimates deterministically.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.