[BUG] A process bug occurred when I tried to use multiple card inference to loop through the prompt words entered by the terminalg

** My Code Follow the official documentation：https://www.deepspeed.ai/tutorials/inference-tutorial/

import os
import torch
import deepspeed
import transofrmers

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

transofrmers.logging.set_verbosity_error()
os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'

local_rank = int(os.environ['LOCAL_RANK','0'])
world_size = int(os.environ['WORLD_SIZE','1'])

MODEL_NAMEorPATH = 'path/to/model'
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAMEorPATH)
model = AutoModelForCausalLM.from_pretrained(MODEL_NAMEorPATH)

generator = pipeline('text-generation', model=model, tokenizer=tokenizer, max_new_tokens=128, device=local_rank)

generator.model = deepspeed.init_inference(generator.model, mp_size=world_size, dtype=torch.float16, replace_with_kernel_inject=True)

while True:
    prompt = input("Model prompt >>> ")
    if prompt == 'quit':
        break
    print(generator(prompt)[0]['generated_text'])

I use the command in the terminal：deepspeed --num_gpus 2 mycode.py
An error occurred, first of all, "Model prompt >>>" was output twice, I checked the related information and code, I learned that when deepspeed uses multi-card infer, it will enable multiple processes, so it will output two, at this time I need to enter prompt twice in the terminal to continue the execution, but there is no output, at this time, both graphics cards are 100%, and the memory does not change. The video memory is also not changing, how should I properly use multi-card multi-thread to properly loop through multiple passes of promp and the correct infer and then get the infer result for each one.

Start with the DeepSpeed inference tutorial and reproduce the example from mycode.py using deepspeed --num_gpus 2 mycode.py. Investigate how the multiple processes handle terminal input and inference, then verify that a prompt is processed once and produces a result without duplicated prompts or a hang.