OptimalScale / OptimalScale/LMFlow

[BUG] did not output the eval results at all.

Open
#815 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.5k
Forks
822
PR merge metrics
No merged PRs in 30d

Description

I went through the process of Finetuning (Full) gpt2, and I set --do_eval --eval_dataset_path xxx.json --eval_steps, where xxx.json is text2text, and the finetune process did not output the eval results at all. My finetune steps exceeded the eval_steps. I don't know if this is a bug or if there is a problem with my settings. I look forward to your answer, thank you very much!
Here is my detailed script setting,

deepspeed ${deepspeed_args}
examples/finetune.py
--model_name_or_path ${model_name_or_path}
--trust_remote_code ${trust_remote_code}
--dataset_path ${dataset_path}
--output_dir ${output_dir} --overwrite_output_dir
--conversation_template ${conversation_template}
--num_train_epochs 0.1
--learning_rate 2e-5
--disable_group_texts 1
--block_size 1024
--per_device_train_batch_size 18
--deepspeed configs/ds_config_zero3.json
--fp16
--run_name finetune
--validation_split_percentage 20
--eval_steps 20
--logging_steps 20
--do_train
--do_eval
--eval_dataset_path /h/s/x/l/eval
--ddp_timeout 72000
--save_steps 5000
--dataloader_num_workers 1
| tee ${log_dir}/train.log
2> ${log_dir}/train.err

Here is the last part of the log during my fine-tuning process,

05/08/2024 10:23:34 - WARNING - lmflow.pipeline.finetuner - finetuner_args.do_evalTrue
05/08/2024 10:23:34 - WARNING - lmflow.pipeline.finetuner - *************************************************************
[2024-05-08 10:23:38,301] [INFO] [partition_parameters.py:326:exit] finished initializing model with 1.64B parameters
05/08/2024 10:23:39 - WARNING - lmflow.pipeline.finetuner - in finetuner_args.do_eval ******************
05/08/2024 10:23:40 - WARNING - lmflow.pipeline.finetuner - ********************************************************************************
05/08/2024 10:23:40 - WARNING - lmflow.pipeline.finetuner - Number of eval samples: 256
ninja: no work to do.
Time to load cpu_adam op: 2.875669002532959 seconds
Parameter Offload: Total persistent parameters: 1001600 in 386 params
{'loss': 0.2962, 'grad_norm': 3.1811087335274615, 'learning_rate': 1.5714285714285715e-05, 'epoch': 0.02}
{'loss': 0.2991, 'grad_norm': 2.5313646679089503, 'learning_rate': 1.0952380952380955e-05, 'epoch': 0.05}
{'loss': 0.3155, 'grad_norm': 2.1892666086453594, 'learning_rate': 6.1904761904761914e-06, 'epoch': 0.07}
{'loss': 0.2972, 'grad_norm': 2.2230829824820884, 'learning_rate': 1.4285714285714286e-06, 'epoch': 0.1}
{'train_runtime': 451.4452, 'train_samples_per_second': 3.316, 'train_steps_per_second': 0.186, 'train_loss': 0.30037035260881695, 'epoch': 0.1}
***** train metrics *****
epoch = 0.101
total_flos = 4229GF
train_loss = 0.3004
train_runtime = 0:07:31.44
train_samples = 14972
train_samples_per_second = 3.316
train_steps_per_second = 0.186

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/finetune.py and the finetuner log output, focusing on how --do_eval, --eval_steps, and --eval_dataset_path are handled during training. Reproduce the command with the provided settings and determine why evaluation metrics are absent despite the log reporting 256 evaluation samples; done means the cause is identified and evaluation results are emitted or the configuration problem is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.