[Usage] How to inference with multi-GPUs in single machine? Possible to do batch inference?

Hi, I have a problem with using Vicuna13b-v1.3 to make an inference with multi-GPU. Could anyone please provide an example of code used for multi-GPU inference without the CLI? On the other hand, is it possible to do batch inference (eg. input a list of prompts and output a list of answers)?

I have tried adjust the num_gpus=2, but seems like it still only compute using single GPU instead of two.
Here is the code

class Vicuna():
    def __init__(self):
        print('Initialize Vicuna...')
        self.model, self.tokenizer = load_model(
            'lmsys/vicuna-13b-v1.3',
            device='cuda',
            num_gpus=2
        )

    @torch.inference_mode()
    def respond(self, input_msg):
        conv = get_conversation_template('lmsys/vicuna-13b-v1.3')
        conv.append_message(conv.roles[0], input_msg)
        conv.append_message(conv.roles[1], None)
        prompt = conv.get_prompt()

        input_ids = self.tokenizer([prompt]).input_ids
        output_ids = self.model.generate(
            torch.as_tensor(input_ids).cuda(),
            do_sample=True,
            temperature=0.001,
            repetition_penalty=1.0,
            max_new_tokens=512,
        )

        output_ids = output_ids[0][len(input_ids[0]) :]
        outputs = self.tokenizer.decode(
            output_ids, skip_special_tokens=True, spaces_between_special_tokens=False
        )
        return outputs


 

        
def main():
    vicuna_model = Vicuna()
    answer= vicuna_model.respond("Who are you?")
    print(answer)


if __name__ == "__main__":
    main()

The issue names load_model and respond but no repository file or test. Start by tracing those entry points and the num_gpus handling; a useful resolution would document non-CLI multi-GPU and batch-inference usage, with verification of GPU use and list inputs.