Budgets exist but meter nothing for self-hosted models
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
`/v1/projects//budget` returns `max_budget_usd`, `spend_usd` and
`remaining_usd`, and `/v1/projects//usage` returns entries with
`total_cost_usd`. The plumbing works.
But **`spend_usd` reads 0.00 regardless of usage.** Self-hosted models have no
entry in LiteLLM's cost map, so every completion is priced at zero. LiteLLM
warns about this on startup:
```
register_model: model=openai/gemma4-26b not in built-in cost map and no
prefix/region variant matched; cache cost fields will default to 0
```
So today the endpoints give **request attribution, not dollars** — and a
project budget can never be exhausted because nothing accrues against it.
Three things to decide:
**What does a self-hosted token cost?** Options: amortised GPU rental over
measured throughput (the A100 is ~$1.19/hr and does ~400 tok/s aggregate, so
roughly $0.0000008/token), a nominal figure for fair-share accounting, or
genuinely zero with budgets used only for rate-limiting.
**Where does it go?** `input_cost_per_token` / `output_cost_per_token` in the
route's `model_info`, set when llmao registers the route. That makes it part of
the same push as the fleet work rather than a separate mechanism.
**What happens at the limit?** LiteLLM blocks the key when a team exceeds its
budget. Worth confirming the failure is legible — a 429 with a clear message
rather than something that reads as an outage.
Note the reporting is only as good as the price. A wrong-but-nonzero figure is
more useful than zero for spotting a runaway job, and less useful than nothing
if anyone treats it as real money. Worth labelling in the UX either way.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the /v1/projects//budget and /v1/projects//usage behavior, then trace how LiteLLM receives input_cost_per_token and output_cost_per_token in the route's model_info. Define a pricing policy for self-hosted tokens and verify that spend accrues, budgets can be exhausted, and the limit response is a clear 429 rather than an outage-like failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100