Alberto-Codes / Alberto-Codes/vramfit

The window pool misprices at a non-default n_ubatch

Open
#433 0 comments 0 reactions 0 assignees View on GitHub
decision
Dominant language
Python
Stars
1
Forks
1
Avg merge
6h 8m
Merged PRs (30d)
105

Description

## The gap

PR #432 (the #431 fix) prices the window pool with
`KV_WINDOW_PAD_TOKENS = 512`, the llama.cpp default `n_ubatch`. The
operator can change `n_ubatch` at serve time (`-ub`). The model then
misprices the pool by `(n_ubatch - 512) x sliding_kv_bytes` per
sequence, silently.

On the Gemma 4 31B fixture one `n_ubatch` step of 512 moves the pool
by 400 MiB per sequence.

## Why #432 did not add a knob

A CLI option or a `KVLayer` field is a decision, not a bug fix. #431
ruled the default-`n_ubatch` figure only. This issue parks the
question: does the padding constant deserve an operator override, and
where does it surface (CLI option, recipe runtime block, or nothing)?

## References

- #431 ruling comment (2026-08-26): "at the default `n_ubatch` 512.
The fixing session decides where the padding constant lives."
- `src/vramfit/domain/budget.py`, `KV_WINDOW_PAD_TOKENS`
- #423 gate comment, step 5 (the 1,200 / 2,400 MiB measurements)

Contributor guide

Open the contributing guide

Research direction

Start in src/vramfit/domain/budget.py at KV_WINDOW_PAD_TOKENS, then read PR #432 and the #431 ruling. Review the #423 gate comment and its 1,200 / 2,400 MiB measurements. Done means deciding whether non-default n_ubatch needs an operator override and where it should surface, then ensuring the pool pricing reflects that decision.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, tooling
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.