lm-sys / lm-sys/FastChat

How to add evaluation?

Open
#2,943 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

I want to train the [vicuna-7b-v1.5] model with the following command:

deepspeed fastchat/train/train_lora.py \
--model_name_or_path lmsys/vicuna-7b-v1.5 \
--lora_r 32 \
--lora_alpha 64 \
--lora_dropout 0.05 \
--data_path ./data/etraindata.json \
--evaluation_data_file ./data/eevaldata.json \
--output_dir ./checkpointsbatch8grad24 \
--num_train_epochs 3 \
--fp16 True \
--per_device_train_batch_size 8 \
--per_device_eval_batch_size 8 \
--gradient_accumulation_steps 16 \
--evaluation_strategy "steps" \
--eval_steps 200 \
--save_strategy "steps" \
--save_steps 200 \
--save_total_limit 2 \
--learning_rate 2e-4 \
--weight_decay 0. \
--warmup_ratio 0.03 \
--lr_scheduler_type "cosine" \
--logging_strategy "steps" \
--logging_steps 1 \
--tf32 True \
--model_max_length 2048 \
--q_lora False \
--deepspeed playground/deepspeed_config_s2.json \
--gradient_checkpointing True \
--flash_attn False \
--report_to wandb \
--run_name "uWu"

but I am getting error as "evaluation_data_file is not recognized by HfArgumentParser"
How can I achieve this?
Also, how to get validation loss?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with train/train_lora.py and inspect the arguments passed to HfArgumentParser, comparing them with the command's evaluation_data_file option. Run the training command with a supported evaluation configuration and verify that validation loss is produced; the issue is resolved when evaluation data and validation loss work without the parser error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.