Account for image inputs in LLM gateway token estimates
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 218
- Forks
- 72
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 427
Description
Problem
The LLM API gateway does not estimate image input tokens from image dimensions
and the target model's visual processor. Multimodal requests can therefore
reserve input-token capacity based on serialized payload size rather than the
visual tokens consumed by inference.
Desired behavior
- Read dimensions from base64-encoded image headers without decoding pixels.
- Bound header inspection so malformed images cannot cause unbounded scans.
- Apply model-specific patch, tile, and fixed-token estimators where known.
- Use a conservative fallback for unknown models or unavailable dimensions.
- Keep model-name matching explicit so unrelated names cannot select a cheaper
estimator.
Acceptance criteria
- Chat Completions and Responses admission accounting includes every image.
- Common NIM vision models use documented processor equations.
- Unknown models receive a size-based bounded estimate.
- PNG, JPEG, GIF, and WebP base64 inputs are covered by tests.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the admission-accounting paths for Chat Completions and Responses and the existing token-estimation and model-name matching logic. Trace how image inputs are represented, then use the acceptance criteria to define completion: bounded header inspection, documented estimators for common NIM vision models, conservative unknown-model handling, and tests for PNG, JPEG, GIF, and WebP base64 inputs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- ai, api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100