apache / apache/tooling-llmao

Budgets exist but meter nothing for self-hosted models

Open
#24 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
1
PR merge metrics
No merged PRs in 30d

Description

`/v1/projects//budget` returns `max_budget_usd`, `spend_usd` and
`remaining_usd`, and `/v1/projects//usage` returns entries with
`total_cost_usd`. The plumbing works.

But **`spend_usd` reads 0.00 regardless of usage.** Self-hosted models have no
entry in LiteLLM's cost map, so every completion is priced at zero. LiteLLM
warns about this on startup:

```
register_model: model=openai/gemma4-26b not in built-in cost map and no
prefix/region variant matched; cache cost fields will default to 0
```

So today the endpoints give **request attribution, not dollars** — and a
project budget can never be exhausted because nothing accrues against it.

Three things to decide:

**What does a self-hosted token cost?** Options: amortised GPU rental over
measured throughput (the A100 is ~$1.19/hr and does ~400 tok/s aggregate, so
roughly $0.0000008/token), a nominal figure for fair-share accounting, or
genuinely zero with budgets used only for rate-limiting.

**Where does it go?** `input_cost_per_token` / `output_cost_per_token` in the
route's `model_info`, set when llmao registers the route. That makes it part of
the same push as the fleet work rather than a separate mechanism.

**What happens at the limit?** LiteLLM blocks the key when a team exceeds its
budget. Worth confirming the failure is legible — a 429 with a clear message
rather than something that reads as an outage.

Note the reporting is only as good as the price. A wrong-but-nonzero figure is
more useful than zero for spotting a runaway job, and less useful than nothing
if anyone treats it as real money. Worth labelling in the UX either way.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the /v1/projects//budget and /v1/projects//usage behavior, then trace how LiteLLM receives input_cost_per_token and output_cost_per_token in the route's model_info. Define a pricing policy for self-hosted tokens and verify that spend accrues, budgets can be exhausted, and the limit response is a clear 429 rather than an outage-like failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.