logits输出有问题[Bug]

### Checklist

- [X] 1. I have searched related issues but cannot get the expected help.
- [X] 2. The bug has not been fixed in the latest version.

### Describe the bug

使用lmdeploy/turbomind/decode.py得到输出的logits，不仅与原模型输出不一致，logits的前半部分解码出来也不是prompt的input_ids
使用的模型是turbomind转换后的internlm2-1.8b的lora合并后的模型

### Reproduction

import os
import os.path as osp

import torch

from lmdeploy import turbomind as tm
from lmdeploy.tokenizer import Tokenizer
from lmdeploy import pipeline, GenerationConfig, PytorchEngineConfig, TurbomindEngineConfig

os.environ['TM_LOG_LEVEL'] = 'ERROR'

resp_label_list = ["A", "B", "C"]

def main(model_path, inputs):
"""An example to perform model inference through the command line
interface.

Args:
model_path (str): the path of the deployed model
inputs (str): the path of text file contatin input text lines
"""
backend_config = TurbomindEngineConfig(
session_len=2048,
cache_max_entry_count=0.01,
)
tokenizer_model_path = osp.join(model_path, 'triton_models', 'tokenizer')
tokenizer = Tokenizer(tokenizer_model_path)
tm_model = tm.TurboMind(model_path, engine_config=backend_config, eos_id=tokenizer.eos_token_id)
generator = tm_model.create_instance()
eval_local_path = "./train.jsonl"

time_cost = 0
correct = 0
max_samples = 500
import json
import time
from tqdm import tqdm
with open(eval_local_path, "r", encoding="utf-8") as file:
for index, line in tqdm(enumerate(file)):
if index >= max_samples:
break
line = line.strip().strip("\n")
obj = json.loads(line)
t1 = time.time()
input_ids = tokenizer.encode([obj["prompt"]])

logits = generator.decode(input_ids)
probs = torch.nn.functional.softmax(
torch.stack(
[logits[:, -1, tokenizer.encode(label)[-1]] for label in resp_label_list],
dim=-1
),
dim=-1
).detach()

offset = torch.argmax(probs, dim=-1)[0]
time_cost += time.time() - t1
correct += obj["response"] == resp_label_list[offset.item()]
assert resp_label_list[offset.item()] == "A"
print({
"index": index,
"infer_score": probs[0][offset].item(),
"infer_label": resp_label_list[offset.item()],
"origin_label": obj["response"],
})

print(f"time: {time_cost}")
print(f"correct: {correct}")

if __name__ == '__main__':

inputs = "请根据给定的文本信息，判断是否存在的违规行为。回复:"
main(model_path="/opt/tiger/gzl/lm_engine/turbomind_output/", inputs=inputs)

### Environment

```Shell
Name: lmdeploy
Version: 0.4.2
Summary: A toolset for compressing, deploying and serving LLM
Home-page:
Author: OpenMMLab
Author-email: openmmlab@gmail.com
License:
Location: /opt/tiger/gzl/gzl_lmdeploy/lib/python3.9/site-packages
Requires: accelerate, einops, fastapi, fire, mmengine-lite, numpy, nvidia-cublas-cu12, nvidia-cuda-runtime-cu12, nvidia-curand-cu12, nvidia-nccl-cu12, peft, pillow, protobuf, pydantic, pynvml, safetensors, sentencepiece, shortuuid, tiktoken, torch, torchvision, transformers, triton, uvicorn
Required-by:
```

### Error traceback

_No response_