Allocate KV cache buffers in pages

**Is your feature request related to a problem? Please describe.**
According to vLLM paper, one issue with LLM inference is the vast differences in memory requirements for different requests, which leads to pre-allocation of memory which is maybe never needed.

For us, the memory is always limited by the KV cache length, but this size is pre-allocated even if the sequence is much shorter.

**Describe the solution you'd like**
At least for applications where many requests need to be served of very different lengths and batch sizes, it would be good to build up KV cache buffers from smaller pages. This is not so useful for training or evaluation on larger datasets, where we'd rather retain KV cache buffers and re-use them (the code does that).

The issue does not name files, tests, or entry points. Start by locating the existing KV cache allocation and reuse code, then understand how request lengths and batch sizes determine buffer size. Done means KV cache buffers can grow from smaller pages for varied inference requests while preserving the existing reuse behavior for training or evaluation.

Written by the indexing model from the issue text.