AnswerDotAI / AnswerDotAI/fastllm

Cost calculation crashes with AttributeError for unmapped models (e.g. local Ollama models)

Open Beginner friendly
#67 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
15
Forks
4
Avg merge
1h 23m
Merged PRs (30d)
12

Description

## Summary

`fastllm`'s cost tracking crashes for model not present in the bundled
`model_prices.json` . Only a small hand-picked subset of Ollama tags
are keys in that file — e.g. `ollama/llama2` is present, but `ministral-3:3b`
(and many other locally-pulled models) isn't.

## Repro

ssage --vendor_name openai_chat --base_url http://localhost:11434/v1
--api_key ollama --model ministral-3:3b "hi"

The completion succeeds, but crashes right after while tracking usage:

File ".../fastllm/openai_chat.py", line 204, in cost
cost = in_txt * m.input_cost_per_token + out_txt * m.output_cost_per_token
AttributeError: input_cost_per_token

## Cause

`cost()` looks up the model name in `model_prices.json` and assumes the
result always has `input_cost_per_token`/`output_cost_per_token`. There's as far as I can see no
fallback for models that aren't a key in that file.

## Suggested fix

Default to `cost = 0` when a model isn't found, rather than raising. Probably
also add a wildcard fallback for ollama models generally.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in fastllm/openai_chat.py around line 204 and inspect how cost() uses the model lookup from model_prices.json. Reproduce with the provided Ollama command and verify that an unmapped model such as ministral-3:3b completes without an AttributeError during usage tracking; determine from the existing lookup whether the wildcard fallback is needed.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, python
Domain
backend
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.