simonw / simonw/llm

[FR] `--json`: Save metadata as JSON

Open
#609 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
12.5k
Forks
998
Avg merge
3d 13h
Merged PRs (30d)
10

Description

I like to track usage costs. Previously, I manually counted the tokens in the inputs and outputs. However, now that models support multimedia input/output, reasoning tokens, and more, my previous approach is no longer feasible. It would be helpful to provide a --json (e.g., --log-json, --json-out, etc.) option that saves the input/output token count and other metadata to a JSON file. This way, my wrapper can read the exact token usage and compute the costs accordingly.

Including the inputs and outputs in this file would also make it useful for logging. In general, this file offers a way to output information without cluttering stdout and hindering interactive, streaming usage.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the command-line option handling and the paths that produce token counts, model metadata, inputs, and outputs. Define the JSON file's contents and how it avoids cluttering stdout, then verify that a wrapper can read exact usage and logging data without disrupting interactive streaming.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, cli
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.