Parameter setting for training Mistral

Nobody has claimed this yet.

Do I need to make any changes to the following training parameters if I am training Mistral?

torchrun --nproc_per_node=1 --master_port=20001 fastchat/train/train_mem.py \
--model_name_or_path /local/Mistral-7B-v0.1 \
--data_path data/dummy_conversation.json \
--bf16 True \
--output_dir result \
--num_train_epochs 3 \
--per_device_train_batch_size 2 \
--per_device_eval_batch_size 2 \
--gradient_accumulation_steps 16 \
--evaluation_strategy "no" \
--save_strategy "steps" \
--save_steps 1200 \
--save_total_limit 10 \
--learning_rate 2e-5 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_steps 1 \
--fsdp "full_shard auto_wrap" \
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
--tf32 True \
--model_max_length 2048 \
--gradient_checkpointing True \
--lazy_preprocess True

The issue provides a torchrun command invoking fastchat/train/train_mem.py with Mistral-7B-v0.1, but names no documentation file or test. Start by checking the training script and existing Mistral or model-specific guidance. Done would require a documented, project-supported answer identifying any parameter changes.

Written by the indexing model from the issue text.