AnswerDotAI / AnswerDotAI/fastllm
Cost calculation crashes with AttributeError for unmapped models (e.g. local Ollama models)
- Dominant language
- Jupyter Notebook
- Stars
- 15
- Forks
- 4
- Avg merge
- 1h 23m
- Merged PRs (30d)
- 12
Description
## Summary
`fastllm`'s cost tracking crashes for model not present in the bundled
`model_prices.json` . Only a small hand-picked subset of Ollama tags
are keys in that file — e.g. `ollama/llama2` is present, but `ministral-3:3b`
(and many other locally-pulled models) isn't.
## Repro
ssage --vendor_name openai_chat --base_url http://localhost:11434/v1
--api_key ollama --model ministral-3:3b "hi"
The completion succeeds, but crashes right after while tracking usage:
File ".../fastllm/openai_chat.py", line 204, in cost
cost = in_txt * m.input_cost_per_token + out_txt * m.output_cost_per_token
AttributeError: input_cost_per_token
## Cause
`cost()` looks up the model name in `model_prices.json` and assumes the
result always has `input_cost_per_token`/`output_cost_per_token`. There's as far as I can see no
fallback for models that aren't a key in that file.
## Suggested fix
Default to `cost = 0` when a model isn't found, rather than raising. Probably
also add a wildcard fallback for ollama models generally.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in fastllm/openai_chat.py around line 204 and inspect how cost() uses the model lookup from model_prices.json. Reproduce with the provided Ollama command and verify that an unmapped model such as ministral-3:3b completes without an AttributeError during usage tracking; determine from the existing lookup whether the wildcard fallback is needed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ollama, python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100