Can't able to get Prediction with Multi-GPU, but working fine in case of Single GPU

Nobody has claimed this yet.

I have the below commands at the start.sh file to start the Vicuna Server, then I'm using OpenAI and LangChain for further uses.
This is working fine with a Single GPU. But When I'm trying to get predictions with Multiple GPUs, I'm facing an issue.

```
source /home/ubuntu/venv_vicuna_311/bin/activate

nohup python3 -m fastchat.serve.controller &

nohup python3 -m fastchat.serve.model_worker --model-names "gpt-3.5-turbo,text-davinci-003,text-embedding-ada-002" --model-path lmsys/vicuna-7b-v1.5 --num-gpus 4 --max-gpu-memory 6GiB &

nohup python3 -m fastchat.serve.openai_api_server --host localhost --port 8080 &

export OPENAI_API_BASE=http://localhost:8080/v1
export OPENAI_API_KEY=EMPTY
```
Also, I have tried --gpus '0,1,2,3' commands. But Nothing seems to work properly.

Then I have tried the below commands mentioned in Readme, that use CLI, but no luck.
`python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3 --num-gpus 4`

Below is a portion of the Error. Full error logs are attached here.

```
python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.3 --num-gpus 4
You are using the default legacy behaviour of the . If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████| 2/2 [01:27<00:00, 43.70s/it]
USER: hi
ASSISTANT: ../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7,0,0], thread: [96,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7,0,0], thread: [97,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7,0,0], thread: [98,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.
../aten/src/ATen/native/cuda/IndexKernel.cu:92: operator(): block: [7,0,0], thread: [99,0,0] Assertion `index >= -sizes[i] && index < sizes[i] && "index out of bounds"` failed.

```

I have verified that the model is loading in multiple GPUs. But the issue occurs when I first attempt to get predictions.

My end goal is to make predictions faster, with the Vicuna model.

[Error Logs Drive Link](https://drive.google.com/file/d/1auqhworHcRdgXDLEy3K7349utPlhlfFw/view?usp=sharing)

Start with the multi-GPU commands in start.sh and the fastchat.serve.model_worker and fastchat.serve.cli entry points. Reproduce the first prediction failure with --num-gpus 4, then inspect the attached CUDA error logs and compare behavior with a single GPU. Done means the Vicuna model can generate predictions through the CLI or OpenAI API when multiple GPUs are enabled.

Written by the indexing model from the issue text.