'CUDA out of memory'. When QLoRA vicuna-7b in 4*24G gps .

hello，I have a question. I get this when using QLoRA.

CUDA out of memory. Tried to allocate 172.00 MiB (GPU 1; 22.38 GiB total capacity; 21.49 GiB already allocated; 62.56 MiB free; 21.69 GiB reserved in total by 
PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
[2023-07-12 17:27:21,958] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 30746
[2023-07-12 17:27:22,399] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 30747
[2023-07-12 17:27:22,426] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 30748
[2023-07-12 17:27:22,442] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 30749
[2023-07-12 17:27:22,442] [ERROR] [launch.py:321:sigkill_handler] ['/home/luojing/miniconda3/envs/vicuna_x/bin/python', '-u', 'fastchat/train/train_lora.py', '--local_rank=3', '--model_name_or_path', '/home/luojing/project/FastChat/vicuna-7b', '--lora_r', '8', '--lora_alpha', '16', '--lora_dropout', '0.05', '--data_path', '/home/luojing/project/FastChat/data/dummy_conversation.json', '--bf16', 'False', '--output_dir', './checkpoints', '--num_train_epochs', '1', '--per_device_train_batch_size', '4', '--per_device_eval_batch_size', '4', '--gradient_accumulation_steps', '1', '--evaluation_strategy', 'no', '--save_strategy', 'steps', '--save_steps', '1200', '--save_total_limit', '100', '--learning_rate', '2e-5', '--weight_decay', '0.', '--warmup_ratio', '0.03', '--lr_scheduler_type', 'cosine', '--logging_steps', '1', '--tf32', 'False', '--model_max_length', '2048', '--q_lora', 'True', '--deepspeed', 'playground/deepspeed_config_s4.json'] exits with return code = 1

deepspeed_config_s4 is as follows

{
  "train_batch_size": "auto",
  "train_micro_batch_size_per_gpu": "auto",
  "gradient_accumulation_steps": "auto",
  "optimizer": {
    "type": "Adam",
    "params": {
      "lr": "auto",
      "weight_decay": "auto"
    }
  },
  "scheduler": {
    "type": "WarmupLR",
    "params": {
      "warmup_min_lr": "auto",
      "warmup_max_lr": "auto",
      "warmup_num_steps": "auto"
    }
  },
  "zero_optimization": {
    "stage": 2,
    "offload_optimizer": {
      "device": "cpu"
    },
    "offload_param": {
      "device": "cpu"
    },
    "overlap_comm": true,
    "contiguous_gradients": true,
    "sub_group_size": 1e9,
    "reduce_bucket_size": "auto",
    "stage3_prefetch_bucket_size": "auto",
    "stage3_param_persistence_threshold": "auto",
    "stage3_max_live_parameters": 1e8,
    "stage3_max_reuse_distance": 1e8,
    "stage3_gather_16bit_weights_on_model_save": true
  },
  "gradient_clipping": 1.0,
  "steps_per_print": 1,
  "bf16": {
    "enabled": false
  },
  "fp16": {
    "loss_scale": 0,
    "auto_cast": true,
    "loss_scale_window": 1000,
    "initial_scale_power": 32,
    "hysteresis": 2,
    "min_loss_scale": 0.5,
    "enabled": false
  }
}

my hardware environment is as follows

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.125.06   Driver Version: 525.125.06   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P40           Off  | 00000000:3B:00.0 Off |                    0 |
| N/A   34C    P0    50W / 250W |      0MiB / 23040MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla P40           Off  | 00000000:86:00.0 Off |                    0 |
| N/A   34C    P0    49W / 250W |      0MiB / 23040MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla P40           Off  | 00000000:AF:00.0 Off |                    0 |
| N/A   31C    P0    49W / 250W |      0MiB / 23040MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla P40           Off  | 00000000:D8:00.0 Off |                    0 |
| N/A   31C    P0    47W / 250W |      0MiB / 23040MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

I am using : Add scaled_dot_product_attention to replace flash attention #177
run command：

deepspeed --num_gpus 4 fastchat/train/train_lora.py \
    --model_name_or_path /home/luojing/project/FastChat/vicuna-7b  \
    --lora_r 8 \
    --lora_alpha 16 \
    --lora_dropout 0.05 \
    --data_path /home/luojing/project/FastChat/data/dummy_conversation.json \
    --bf16 False \
    --output_dir ./checkpoints \
    --num_train_epochs 1 \
    --per_device_train_batch_size 4 \
    --per_device_eval_batch_size 4 \
    --gradient_accumulation_steps 1 \
    --evaluation_strategy "no" \
    --save_strategy "steps" \
    --save_steps 1200 \
    --save_total_limit 100 \
    --learning_rate 2e-5 \
    --weight_decay 0. \
    --warmup_ratio 0.03 \
    --lr_scheduler_type "cosine" \
    --logging_steps 1 \
    --tf32 False \
    --model_max_length 2048 \
    --q_lora True \
    --deepspeed playground/deepspeed_config_s4.json

I think it might be because ddp = world_size!=1 is true, so model.is_parallelizable = True and model.model_parallel = True didn't work.
I'm not sure how model parallelism should be done to enable me to do qlora fine-tuning on multiple gpu's.
By the way, Is it possible to do qlora fine-tuning of vicuna-33b in this hardware environment ？
I need some help,Thanks.

Originally posted in https://github.com/lm-sys/FastChat/issues/1900#issuecomment-1632206925