Fine-tuning Vicuna-7B with Local GPUs

Nobody has claimed this yet.

torchrun --nproc_per_node=4 --master_port=20001 /raid/users/lifei/FastChat/fastchat/train/train_mem.py --model_name_or_path /raid/users/mrh/weights/vicuna-7b --data_path /raid/users/lifei/FastChat/playground/data/dummy.json --bf16 True --output_dir output --num_train_epochs 3 --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --gradient_accumulation_steps 16 --evaluation_strategy "no" --save_strategy "steps" --save_steps 1200 --save_total_limit 10 --learning_rate 2e-5 --weight_decay 0. --warmup_ratio 0.03 --lr_scheduler_type "cosine" --logging_steps 1 --fsdp "full_shard auto_wrap" --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' --tf32 True --model_max_length 2048 --gradient_checkpointing True --lazy_preprocess True

RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Start with the provided torchrun command and fastchat/train/train_mem.py, using CUDA_LAUNCH_BLOCKING=1 as suggested to obtain a more accurate failing operation. Check the reported CUDA error against the four-GPU, FSDP and LlamaDecoderLayer configuration. Done means identifying a reproducible cause and documenting or correcting the configuration that triggers the invalid argument.

Written by the indexing model from the issue text.