difference during inference with or without kvcache

Nobody has claimed this yet.

### Bug description

i modified the generate/base.py , one inference with kvcache, and the other one is without kvcache,

i set both temperature=0, topk=None and topp=0,
i use the same seed,
i use the same model(qwen2.5-0.5b-instruct)

the only thing i did is in the function **generate_fn**,
-------------------------------
if prefill_token:
tmp_x = token.view(1, -1)
else:
tmp_x = torch.cat(all_tokens, dim=0).view(1, -1)
token = next_token(
model,
input_pos=None,
x=tmp_x,
input_pos_maxp1=None,
temperature=temperature,
top_k=top_k,
top_p=top_p,
prefill_token=prefill_token,
count=count,
)
-------------------------------

but i found that the logits is difference,

im not sure if the difference is ok,

### Reproduced in studio

_No response_

### What operating system are you using?

Unknown

### LitGPT Version

_No response_

Start in generate/base.py at generate_fn and the next_token call shown in the report. Compare the logits produced with and without KV cache using the same model, seed, and sampling settings, and determine whether the difference is expected. Done means the cause or expected behavior is documented and covered by a reproducible check.

Written by the indexing model from the issue text.