Alberto-Codes / Alberto-Codes/vramfit
The window pool misprices at a non-default n_ubatch
- Dominant language
- Python
- Stars
- 1
- Forks
- 1
- Avg merge
- 6h 8m
- Merged PRs (30d)
- 105
Description
## The gap
PR #432 (the #431 fix) prices the window pool with
`KV_WINDOW_PAD_TOKENS = 512`, the llama.cpp default `n_ubatch`. The
operator can change `n_ubatch` at serve time (`-ub`). The model then
misprices the pool by `(n_ubatch - 512) x sliding_kv_bytes` per
sequence, silently.
On the Gemma 4 31B fixture one `n_ubatch` step of 512 moves the pool
by 400 MiB per sequence.
## Why #432 did not add a knob
A CLI option or a `KVLayer` field is a decision, not a bug fix. #431
ruled the default-`n_ubatch` figure only. This issue parks the
question: does the padding constant deserve an operator override, and
where does it surface (CLI option, recipe runtime block, or nothing)?
## References
- #431 ruling comment (2026-08-26): "at the default `n_ubatch` 512.
The fixing session decides where the padding constant lives."
- `src/vramfit/domain/budget.py`, `KV_WINDOW_PAD_TOKENS`
- #423 gate comment, step 5 (the 1,200 / 2,400 MiB measurements)
Contributor guide
Research direction
Start in src/vramfit/domain/budget.py at KV_WINDOW_PAD_TOKENS, then read PR #432 and the #431 ruling. Review the #423 gate comment and its 1,200 / 2,400 MiB measurements. Done means deciding whether non-default n_ubatch needs an operator override and where it should surface, then ensuring the pool pricing reflects that decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, tooling
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100