Inference for ctranslate2 using tensor parallel with mpi

import ctranslate2,psutil,os,transformers,time,torch

generator = ctranslate2.Generator("/ct2opt-1.3b",tensor_parallel=True,device="cuda")
tokenizer = transformers.AutoTokenizer.from_pretrained("facebook/opt-1.3b")

def generate_text(text):
for prompt in text:
start_tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(prompt))
results = generator.generate_batch([start_tokens], max_length=30,include_prompt_in_result=False)
output = tokenizer.decode(results[0].sequences_ids[0])
return output

text = ["Hello, I am"]
results=generate_text(text)
print(results)

i am getting 4 outputs when i run this script using this command: mpirun -np 4 python3 ffctranslateload.py
and the results are very bad. although the model is distributed.when i keep -np 1 the results are good. how to get good results when i keep -np 4

Start by running ffctranslateload.py with mpirun -np 1 and -np 4, using the shown ctranslate2.Generator configuration and facebook/opt-1.3b tokenizer. Investigate why four outputs differ and whether the tensor-parallel setup is being used correctly; done means reproducible, good-quality generation with four processes.