CUDA Out-of-memory when fine-tuning Vicuna-13B-v1.5 with QLoRA

Nobody has claimed this yet.

# Command
```
deepspeed fastchat/train/train_lora.py \
--model_name_or_path ./vicuna-13b-v1.5 \
--lora_r 2 \
--lora_alpha 16 \
--lora_dropout 0.05 \
--data_path data.json \
--bf16 True \
--output_dir ./checkpoints \
--num_train_epochs 3 \
--per_device_train_batch_size 1 \
--per_device_eval_batch_size 1 \
--gradient_accumulation_steps 1 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 1200 \
--save_total_limit 100 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--tf32 True \
--model_max_length 4096 \
--q_lora True \
--deepspeed ../playground/deepspeed_config_s2.json
```

# Warning
```
[2023-11-20 12:29:46,034] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-11-20 12:29:46,897] [INFO] [launch.py:145:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2023-11-20 12:29:46,897] [INFO] [launch.py:151:main] nnodes=1, num_local_procs=2, node_rank=0
[2023-11-20 12:29:46,897] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(, {'localhost': [0, 1]})
[2023-11-20 12:29:46,897] [INFO] [launch.py:163:main] dist_world_size=2
[2023-11-20 12:29:46,897] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0,1
[2023-11-20 12:29:48,314] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-11-20 12:29:48,339] [INFO] [real_accelerator.py:110:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2023-11-20 12:29:50,642] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2023-11-20 12:29:50,643] [INFO] [comm.py:594:init_distributed] cdb=None
[2023-11-20 12:29:50,643] [INFO] [comm.py:625:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2023-11-20 12:29:50,837] [WARNING] [comm.py:152:init_deepspeed_backend] NCCL backend in DeepSpeed not yet implemented
[2023-11-20 12:29:50,837] [INFO] [comm.py:594:init_distributed] cdb=None
Loading checkpoint shards: 100%|█████████████████████████████████████████████████| 3/3 [00:18<00:00, 6.08s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████| 3/3 [00:19<00:00, 6.66s/it]
trainable params: 1,638,400 || all params: 6,673,617,920 || trainable%: 0.024550401590866022
[WARNING] cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!
Using /home/pym/.cache/torch_extensions/py310_cu117 as PyTorch extensions root...
Emitting ninja build file /home/pym/.cache/torch_extensions/py310_cu117/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.4718854427337646 seconds
[WARNING] cpu_adam cuda is missing or is incompatible with installed torch, only cpu ops can be compiled!
Using /home/pym/.cache/torch_extensions/py310_cu117 as PyTorch extensions root...
Emitting ninja build file /home/pym/.cache/torch_extensions/py310_cu117/cpu_adam/build.ninja...
Building extension module cpu_adam...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
ninja: no work to do.
Loading extension module cpu_adam...
Time to load cpu_adam op: 2.5696916580200195 seconds
Rank: 1 partition count [2] and sizes[(819200, False)]
Rank: 0 partition count [2] and sizes[(819200, False)]
```

# Error

```
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /media/pym/date/Train-FastChat/Train-FastChat/fastchat/train/train_lora.py:222 in │
│ │
│ 219 │
│ 220 │
│ 221 if __name__ == "__main__": │
│ ❱ 222 │ train() │
│ 223 │
│ │
│ /media/pym/date/Train-FastChat/Train-FastChat/fastchat/train/train_lora.py:198 in train │
│ │
│ 195 │ if list(pathlib.Path(training_args.output_dir).glob("checkpoint-*")): │
│ 196 │ │ trainer.train(resume_from_checkpoint=True) │
│ 197 │ else: │
│ ❱ 198 │ │ trainer.train() │
│ 199 │ trainer.save_state() │
│ 200 │ │
│ 201 │ # check if zero3 mode enabled │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/transformers/trainer.py:1645 in │
│ train │
│ │
│ 1642 │ │ inner_training_loop = find_executable_batch_size( │
│ 1643 │ │ │ self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size │
│ 1644 │ │ ) │
│ ❱ 1645 │ │ return inner_training_loop( │
│ 1646 │ │ │ args=args, │
│ 1647 │ │ │ resume_from_checkpoint=resume_from_checkpoint, │
│ 1648 │ │ │ trial=trial, │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/transformers/trainer.py:1929 in │
│ _inner_training_loop │
│ │
│ 1926 │ │ │ │ │ self.control = self.callback_handler.on_step_begin(args, self.state, │
│ 1927 │ │ │ │ │
│ 1928 │ │ │ │ with self.accelerator.accumulate(model): │
│ ❱ 1929 │ │ │ │ │ tr_loss_step = self.training_step(model, inputs) │
│ 1930 │ │ │ │ │
│ 1931 │ │ │ │ if ( │
│ 1932 │ │ │ │ │ args.logging_nan_inf_filter │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/transformers/trainer.py:2750 in │
│ training_step │
│ │
│ 2747 │ │ │ return loss_mb.reduce_mean().detach().to(self.args.device) │
│ 2748 │ │ │
│ 2749 │ │ with self.compute_loss_context_manager(): │
│ ❱ 2750 │ │ │ loss = self.compute_loss(model, inputs) │
│ 2751 │ │ │
│ 2752 │ │ if self.args.n_gpu > 1: │
│ 2753 │ │ │ loss = loss.mean() # mean() to average on multi-gpu parallel training │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/transformers/trainer.py:2775 in │
│ compute_loss │
│ │
│ 2772 │ │ │ labels = inputs.pop("labels") │
│ 2773 │ │ else: │
│ 2774 │ │ │ labels = None │
│ ❱ 2775 │ │ outputs = model(**inputs) │
│ 2776 │ │ # Save past state if it exists │
│ 2777 │ │ # TODO: this needs to be fixed and made cleaner later. │
│ 2778 │ │ if self.args.past_index >= 0: │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/deepspeed/utils/nvtx.py:15 in │
│ wrapped_fn │
│ │
│ 12 │ │
│ 13 │ def wrapped_fn(*args, **kwargs): │
│ 14 │ │ get_accelerator().range_push(func.__qualname__) │
│ ❱ 15 │ │ ret_val = func(*args, **kwargs) │
│ 16 │ │ get_accelerator().range_pop() │
│ 17 │ │ return ret_val │
│ 18 │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/deepspeed/runtime/engine.py:1735 │
│ in forward │
│ │
│ 1732 │ │ if self.fp16_auto_cast(): │
│ 1733 │ │ │ inputs = self._cast_inputs_half(inputs) │
│ 1734 │ │ │
│ ❱ 1735 │ │ loss = self.module(*inputs, **kwargs) │
│ 1736 │ │ │
│ 1737 │ │ if self.zero_optimization_partition_weights(): │
│ 1738 │ │ │ # Disable automated discovery of external parameters │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/peft/peft_model.py:857 in forward │
│ │
│ 854 │ │ │ │ │ **kwargs, │
│ 855 │ │ │ │ ) │
│ 856 │ │ │ │
│ ❱ 857 │ │ │ return self.base_model( │
│ 858 │ │ │ │ input_ids=input_ids, │
│ 859 │ │ │ │ attention_mask=attention_mask, │
│ 860 │ │ │ │ inputs_embeds=inputs_embeds, │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/accelerate/hooks.py:165 in │
│ new_forward │
│ │
│ 162 │ │ │ with torch.no_grad(): │
│ 163 │ │ │ │ output = old_forward(*args, **kwargs) │
│ 164 │ │ else: │
│ ❱ 165 │ │ │ output = old_forward(*args, **kwargs) │
│ 166 │ │ return module._hf_hook.post_forward(module, output) │
│ 167 │ │
│ 168 │ module.forward = new_forward │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/transformers/models/llama/modeling │
│ _llama.py:688 in forward │
│ │
│ 685 │ │ return_dict = return_dict if return_dict is not None else self.config.use_return │
│ 686 │ │ │
│ 687 │ │ # decoder outputs consists of (dec_features, layer_state, dec_hidden, dec_attn) │
│ ❱ 688 │ │ outputs = self.model( │
│ 689 │ │ │ input_ids=input_ids, │
│ 690 │ │ │ attention_mask=attention_mask, │
│ 691 │ │ │ position_ids=position_ids, │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/accelerate/hooks.py:165 in │
│ new_forward │
│ │
│ 162 │ │ │ with torch.no_grad(): │
│ 163 │ │ │ │ output = old_forward(*args, **kwargs) │
│ 164 │ │ else: │
│ ❱ 165 │ │ │ output = old_forward(*args, **kwargs) │
│ 166 │ │ return module._hf_hook.post_forward(module, output) │
│ 167 │ │
│ 168 │ module.forward = new_forward │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/transformers/models/llama/modeling │
│ _llama.py:578 in forward │
│ │
│ 575 │ │ │ │ │ None, │
│ 576 │ │ │ │ ) │
│ 577 │ │ │ else: │
│ ❱ 578 │ │ │ │ layer_outputs = decoder_layer( │
│ 579 │ │ │ │ │ hidden_states, │
│ 580 │ │ │ │ │ attention_mask=attention_mask, │
│ 581 │ │ │ │ │ position_ids=position_ids, │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/accelerate/hooks.py:165 in │
│ new_forward │
│ │
│ 162 │ │ │ with torch.no_grad(): │
│ 163 │ │ │ │ output = old_forward(*args, **kwargs) │
│ 164 │ │ else: │
│ ❱ 165 │ │ │ output = old_forward(*args, **kwargs) │
│ 166 │ │ return module._hf_hook.post_forward(module, output) │
│ 167 │ │
│ 168 │ module.forward = new_forward │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/transformers/models/llama/modeling │
│ _llama.py:292 in forward │
│ │
│ 289 │ │ hidden_states = self.input_layernorm(hidden_states) │
│ 290 │ │ │
│ 291 │ │ # Self Attention │
│ ❱ 292 │ │ hidden_states, self_attn_weights, present_key_value = self.self_attn( │
│ 293 │ │ │ hidden_states=hidden_states, │
│ 294 │ │ │ attention_mask=attention_mask, │
│ 295 │ │ │ position_ids=position_ids, │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in │
│ _call_impl │
│ │
│ 1498 │ │ if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks │
│ 1499 │ │ │ │ or _global_backward_pre_hooks or _global_backward_hooks │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ 1503 │ │ full_backward_hooks, non_full_backward_hooks = [], [] │
│ 1504 │ │ backward_pre_hooks = [] │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/accelerate/hooks.py:165 in │
│ new_forward │
│ │
│ 162 │ │ │ with torch.no_grad(): │
│ 163 │ │ │ │ output = old_forward(*args, **kwargs) │
│ 164 │ │ else: │
│ ❱ 165 │ │ │ output = old_forward(*args, **kwargs) │
│ 166 │ │ return module._hf_hook.post_forward(module, output) │
│ 167 │ │
│ 168 │ module.forward = new_forward │
│ │
│ /home/pym/anaconda3/envs/chatbot/lib/python3.10/site-packages/transformers/models/llama/modeling │
│ _llama.py:212 in forward │
│ │
│ 209 │ │ │
│ 210 │ │ past_key_value = (key_states, value_states) if use_cache else None │
│ 211 │ │ │
│ ❱ 212 │ │ attn_weights = torch.matmul(query_states, key_states.transpose(2, 3)) / math.sqr │
│ 213 │ │ │
│ 214 │ │ if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len): │
│ 215 │ │ │ raise ValueError( │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
OutOfMemoryError: CUDA out of memory. Tried to allocate 1.25 GiB (GPU 1; 79.10 GiB total capacity; 76.13 GiB
already allocated; 290.12 MiB free; 77.47 GiB reserved in total by PyTorch) If reserved memory is >> allocated
memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and
PYTORCH_CUDA_ALLOC_CONF

```

Start with fastchat/train/train_lora.py and the supplied DeepSpeed command, then inspect the training path around train() and the reported model, sequence length, and two-GPU setup. Reproduce the CUDA out-of-memory failure and identify a project change that prevents it, with a test or documented successful run showing that fine-tuning completes.

Written by the indexing model from the issue text.