How to use multi GPU to finetune Llama2

Hi, I have a question about how to finetune Llama2 by using multi-GPU.

env: 4*A100 40G
yaml: llm/vaseline/exp_yaml/dolly_lda/dolly_federate.yaml

this yaml likes as follow
```
use_gpu: True
device: 0
early_stop:
patience: 0
federate:
mode: standalone
client_num: 3
total_round_num: 500
```
**only one** A100 is not enough, how can I use **other three GPUS** to finetune my model.

I try to modify `train.data_para_dids=[0, 1, 2, 3]`, but it is not work, i think the reason is `cfg.device` only specify one GPU.

Wish your reply!

Start with llm/vaseline/exp_yaml/dolly_lda/dolly_federate.yaml and trace how cfg.device and train.data_para_dids are consumed. Check the Llama2 fine-tuning entry point and run the configuration on the stated four-GPU environment. Done means the supported multi-GPU setup, or its limitation, is clearly documented.

Written by the indexing model from the issue text.