[Bug]

### Checklist

- [ ] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.
- [ ] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

这个 commit ：https://github.com/InternLM/lmdeploy/commit/14b6c02a8f9262d5c4e811cbb504bbff68dfcf95#diff-55ddd52837f63ac13fc0c56158455fe3e47d5650784a1fc40230d4ab93a009f0 后：

调用 llama_triton_example 离线推理的时候，模型会运行到这里报错：
，看起来是没有调用prefix caching却走到了prefix caching的代码里。

运行的模型 Config：
[llama]
model_name = llama
model_arch = LlamaForCausalLM
tensor_para_size = 1
head_num = 32
kv_head_num = 8
vocab_size = 128256
num_layer = 32
inter_size = 14336
norm_eps = 1e-05
attn_bias = 0
start_id = 128000
end_id = 128009
session_len = 131080
weight_type = bf16
rotary_embedding = 128
rope_theta = 500000.0
size_per_head = 128
group_size = 0
max_batch_size = 64
max_context_token_num = 1
step_length = 1
cache_max_entry_count = 0.8
cache_block_seq_len = 64
cache_chunk_size = -1
enable_prefix_caching = False
num_tokens_per_iter = 0
max_prefill_iters = 1
extra_tokens_per_iter = 0
use_context_fmha = 1
quant_policy = 0
max_position_embeddings = 131072
original_max_position_embeddings = 8192
rope_scaling_type = llama3
rope_scaling_factor = 8.0
use_dynamic_ntk = 0
low_freq_factor = 1.0
high_freq_factor = 4.0
use_logn_attn = 0
lora_policy =
lora_r = 0
lora_scale = 0.0
lora_max_wo_r = 0
lora_rank_pattern =
lora_scale_pattern =
__pydantic_initialised__ = True

### Reproduction

-

### Environment

```Shell
-
```

### Error traceback

```Shell
-
```