AnswerDotAI / AnswerDotAI/fastllm

AttributeError: input_cost_per_token when using an unregistered model (e.g. Fireworks DeepSeek V4 Flash)

Open
#86 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
15
Forks
4
Avg merge
1h 23m
Merged PRs (30d)
12

Description

# Bug report: `AttributeError: input_cost_per_token` when using an unregistered model (e.g. Fireworks DeepSeek V4 Flash)

**Repo:** AnswerDotAI/fastllm
**Type:** Bug
**Severity:** Medium — crashes at the end of *every* turn for any model missing cost metadata, after the response has already been delivered.

## Summary

Calling `acomplete` / `AsyncChat` with a valid vendor-prefixed model that is **not** present in either `model_info_registry` (`fastllm/types.py`) or the bundled litellm price DB (`fastllm/model_prices.json`) succeeds over the wire, streams the full response, and then crashes while tracking usage with:

```
AttributeError: input_cost_per_token
```

The metadata lookup returns an **empty** `AttrDict` (via `dict2obj({})`), and the API-specific `cost()` functions access pricing fields by attribute, e.g. `openai_chat.py`:

```python
cost = in_txt * m.input_cost_per_token + out_txt * m.output_cost_per_token
```

`AttrDict` raises `AttributeError` on missing attributes (`.get()` works fine, attribute access does not).

## Concrete example

Model: `accounts/fireworks/models/deepseek-v4-flash-0731` via `fireworks_ai/` prefix — a **valid, working** model on the Fireworks API (the unversioned `deepseek-v4-flash` ID returns 404). It is absent from both metadata sources, so the crash fires on completion.

Verified with fastllm 0.0.41 + shell_sage 1.1.1; same code exists on `main` (checked 2026-08-17).

## Minimal reproduction (no API key required)

```python
from fastllm.types import get_model_info, Usage
from fastllm.openai_chat import cost

meta = get_model_info('accounts/fireworks/models/deepseek-v4-flash-0731', 'fireworks_ai')
assert not dict(meta) # metadata lookup misses

cost(Usage(prompt_tokens=10, completion_tokens=5,
raw={'prompt_tokens': 10, 'completion_tokens': 5}), meta)
# AttributeError: input_cost_per_token
```

End-to-end path (requires an API key): `AsyncChat._call` → `self._track(res)` → `UsageStats.from_response(res)` → `cost=r.cost` → `Completion.cost` property (`types.py`) → `api.cost(self.usage, meta)`.

## Root cause

- `get_model_info(mn, vendor_name)` (`types.py`) falls back to `get_model_meta`, which returns `dict2obj({})` when the model is in neither `model_info_registry` nor litellm's `model_prices.json`.
- `cost()` in **all four** API modules then does attribute access on that empty object:
- `openai_chat.py:202` — `m.input_cost_per_token` etc.
- `openai_responses.py:251` — via `tier_rate(m, 'input_cost_per_token', tier)`
- `anthropic.py:336` — `m.input_cost_per_token`
- `gemini.py:274` — via `tier_rate(m, ...)` (this one is partially guarded since `tier_rate` uses `meta.get(...)`, but `approx_pricing` in `types.py` does `p['input_cost_per_token']` → `KeyError` on empty dict)
- The codebase is already defensive *elsewhere* against missing metadata — e.g. `_prep_call` uses `model_info.get('max_output_tokens', 32_000)` — but `cost()` is not.

## Impact

- Any model that is valid but unregistered crashes at the end of each turn, **after** the response has fully streamed — confusing for users (looks like the answer caused an error), and exit code becomes non-zero.
- This includes common hosted models whose IDs are versioned (e.g. `-0731`) and thus never match the price DB.
- `max_output_tokens` also silently degrades to the 32k fallback for these models.

## Suggested fix

Make cost computation defensive when metadata is missing. Either:

1. In `types.py`, `Completion.cost` property: if the resolved metadata dict is empty, return `0.0` (or skip cost entirely):

```python
meta = get_model_info(self.model, self.vendor_name)
if not dict(meta): return 0.0
```

2. And/or switch the per-API `cost()` implementations to `.get(..., 0)`/`tier_rate()` guards so a partial metadata dict can't crash either.

## Workaround (until fixed)

Register the model via fastllm's public API at startup:

```python
from fastllm.types import register_model_info, modern_llm
register_model_info(
'accounts/fireworks/models/deepseek-v4-flash-0731', vendor_name='fireworks_ai', **modern_llm,
max_input_tokens=1048576, max_output_tokens=384000, max_tokens=384000,
input_cost_per_token=1.4e-07, output_cost_per_token=2.8e-07,
cache_read_input_token_cost=2.8e-08,
)
```

(pricing values taken from the bundled litellm entry for `fireworks_ai/accounts/fireworks/models/deepseek-v4-flash`)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with get_model_info and Completion.cost in fastllm/types.py, then inspect cost() in openai_chat.py, openai_responses.py, anthropic.py, and gemini.py. Run the minimal reproduction from the issue and add or update coverage for an unregistered model. Done means missing or partial cost metadata no longer raises during cost tracking after a response completes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.