[Bug] The program on the main graphics card gets stuck when running with multiple graphics cards.

### Checklist

- [x] 1. I have searched related issues but cannot get the expected help.
- [x] 2. The bug has not been fixed in the latest version.
- [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.

### Describe the bug

When I run the 78B model using lmdeploy with 8 A100, I find that the program on the main graphics card will be terminated. However, the utilization rate of the other seven graphics cards remains at 100%, and the program does not report any errors. The llmdeploy ==0.7.0 .

### Reproduction
```python
print('*** loading model ***')
gen_config = GenerationConfig(temperature=0,max_new_tokens=3500)
print(GenerationConfig())
print(TurbomindEngineConfig())
# model = '/group/40048/elbertzhang/data/data/models/InternVL2_5/InternVL2_5-78B-MPO-AWQ'
model = "/group/40048/keningliu/tools/models/InternVL2_5-78B-MPO"
# model = "/group/40048/elbertzhang/data/data/models/InternVL2_5/InternVL2_5-8B-MPO"
pipe = pipeline(model, backend_config=TurbomindEngineConfig(session_len=12000, tp=8, cache_max_entry_count=0.95))

print('*** loading data ***')
folder_path = '/group/40048/elbertzhang/data/data/scorer/MMPR' # 替换为你的文件夹路径
data_mmpr = []
for filename in tqdm(os.listdir(folder_path)):
if (filename.startswith('data_v1_InternVL2_5_batch_20000_') or filename.startswith('data_v1_InternVL2_5_batch_5000_')) and filename.endswith('.json'):

file_path = os.path.join(folder_path, filename)
data_tmp = load_json(file_path)
data_mmpr += data_tmp

print('*** processing ***')

# # single image
# for item in tqdm(data_debug):
# try:
# question = item['question']
# chosen = item['chosen']
# rejected = item['rejected']
# img_root = item['img_root']
# dataset_name = item["dataset_name"]

# tmp_dict = {}
# item_info = item

# img_path = os.path.join('/group/40048/public', img_root, item_info["image"])

# pixel_values = load_image(img_path, max_num=12).to(torch.bfloat16).cuda()
# generation_config = dict(max_new_tokens=2024, do_sample=False)

# flag = choice(['A','B'])

# if flag=='A':
# prompt = f"You are an intelligent assistant skilled in multimodal content understanding and analysis. I'll give you the image and the corresponding question, along with two possible responses to that question, and I'll prompt you on which response is better, and your task is to use the prompt to explain in detail why that response is better.\n\nImage: \n\n[Beginning of the user's question]\n{question}\n[End of the user's question]\n\n[Beginning of Model A's answer]\n{chosen}\n[End of Model A's answer]\n\n[Beginning of Model B's answer]\n{rejected}\n[End of Model B's answer]\n\nThe better response: [Model A]\n\nAnalyze:"
# else:
# prompt = f"You are an intelligent assistant skilled in multimodal content understanding and analysis. I'll give you the image and the corresponding question, along with two possible responses to that question, and I'll prompt you on which response is better, and your task is to use the prompt to explain in detail why that response is better.\n\nImage: \n\n[Beginning of the user's question]\n{question}\n[End of the user's question]\n\n[Beginning of Model A's answer]\n{rejected}\n[End of Model A's answer]\n\n[Beginning of Model B's answer]\n{chosen}\n[End of Model B's answer]\n\nThe better response: [Model B]\n\nAnalyze:"

# response = model.chat(tokenizer, pixel_values, prompt, generation_config)

# tmp_dict["dataset_name"] = dataset_name
# tmp_dict["image"] = img_path
# tmp_dict["analyze"] = response
# tmp_dict["question"] = question
# tmp_dict["chosen"] = chosen
# tmp_dict["rejected"] = rejected
# tmp_dict["flag"] = flag

# data_vllm.append(tmp_dict)

# except Exception as e:
# print(f"在处理数据 {item} 时出现错误，错误信息: {str(e)}")
# continue


# batch
n = 4
index=0
generate_output = []
for i in tqdm(range(0, len(data_mmpr), n)):
chunk_ori = data_mmpr[i:i+n] # 切片获取当前4个元素
chunk = copy.deepcopy(chunk_ori)
try:
input_str=[]
for item_info in chunk:
question = item_info['question'].replace('', '')
flag = item_info["flag"]
if flag=='A':
prompts = f"You are an intelligent assistant skilled in multimodal content understanding and analysis. I'll give you the image and the corresponding question, along with two possible responses to that question, and I'll prompt you on which response is better, and your task is to use the prompt to explain in detail why that response is better.\n\n[Beginning of the user's question]\n{question}\n[End of the user's question]\n\n[Beginning of Model A's answer]\n{item_info['chosen']}\n[End of Model A's answer]\n\n[Beginning of Model B's answer]\n{item_info['rejected']}\n[End of Model B's answer]\n\nThe better response: [Model A]\n\nAnalyze:"
else:
prompts = f"You are an intelligent assistant skilled in multimodal content understanding and analysis. I'll give you the image and the corresponding question, along with two possible responses to that question, and I'll prompt you on which response is better, and your task is to use the prompt to explain in detail why that response is better.\n\n[Beginning of the user's question]\n{question}\n[End of the user's question]\n\n[Beginning of Model A's answer]\n{item_info['rejected']}\n[End of Model A's answer]\n\n[Beginning of Model B's answer]\n{item_info['chosen']}\n[End of Model B's answer]\n\nThe better response: [Model B]\n\nAnalyze:"
image_path = item_info["img_path"]
input_str.append((prompts,load_image(image_path)))
# print(input_str)
responses = pipe(input_str, gen_config=gen_config)

for question_item, response in zip(chunk,responses):
tmp_dict = {}
question = question_item['question'].replace('', '')
tmp_dict["image"] = question_item["img_path"]
tmp_dict["conversations"] = []
human_value = f"You are provided with an image and a question for this image. Please review the corresponding responses based on the following 5 factors: 1. Accuracy in Object Description: Evaluate the accuracy of the descriptions concerning the objects mentioned in the ground truth answer. Responses should minimize the mention of objects not present in the ground truth answer, and inaccuracies in the description of existing objects. 2. Accuracy in Depicting Relationships: Consider how accurately the relationships between objects are described compared to the ground truth answer. Rank higher the responses that least misrepresent these relationships. 3. Accuracy in Describing Attributes: Assess the accuracy in the depiction of objects' attributes compared to the ground truth answer. Responses should avoid inaccuracies in describing the characteristics of the objects present. 4. Helpfulness: Consider whether the generated text provides valuable insights, additional context, or relevant information that contributes positively to the user's comprehension of the image. Assess whether the language model accurately follows any specific instructions or guidelines provided in the prompt. Evaluate the overall contribution of the response to the user experience. 5. Ethical Considerations: - Identify if the model gives appropriate warnings or avoids providing advice on sensitive topics, such as medical images. Ensure the model refrains from stating identification information in the image that could compromise personal privacy. Evaluate the language model's responses for fairness in treating individuals and communities, avoiding biases. Assess for harmfulness, ensuring the avoidance of content that may potentially incite violence, be classified as NSFW (Not Safe For Work), or involve other unmentioned ethical considerations. Consider any content that could be deemed offensive, inappropriate, or ethically problematic beyond the explicitly listed criteria. You need to choose which response is better for the given question and provide a detailed reason.\nYour task is provided as follows:\n[Beginning of the user's question]\n{question}\n[End of the user's question]\n\n[Beginning of Model A's answer]\n{question_item['rejected']}\n[End of Model A's answer]\n\n[Beginning of Model B's answer]\n{question_item['chosen']}\n[End of Model B's answer]\n"
human ={"from": "human","value" : human_value}
ai_value = response.text
ai ={"from": "gpt","value" : ai_value}
tmp_dict["conversations"].append(human)
tmp_dict["conversations"].append(ai)
generate_output.append(tmp_dict)
except Exception as e:
print(f"在处理数据 {chunk} 时出现错误，错误信息: {str(e)}")
continue


file_path = f'/group/40048/keningliu/tools/tmp_data/MPO78B_mmpr87921.json'

# 使用json.dump将字典数据保存为JSON文件
with open(file_path, 'w') as f:
json.dump(generate_output, f,indent=4)

print(f'*** save to {file_path} ***')

```
### Environment

```Shell
llmdeploy=0.7.0
```

### Error traceback

```Shell

```