lm-sys / lm-sys/FastChat

How to get logs for every x number of steps instead of epochs

Open
#2,528 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
39.5k
Forks
4.8k
PR merge metrics
No merged PRs in 30d

Description

Right now we are getting logs for each 0.5 epoch but I want logs for each x number of steps. how i can achieve this

Description:
I am currently using the training script and noticed that the logs are generated every 0.25 epochs. For my use-case, it would be more beneficial to get logs every x number of steps. This granularity would help in better monitoring and understanding of the training dynamics over shorter intervals.

Screenshot 2023-10-09 at 7 14 13 AM

Details:

Current Behavior: Logs are generated every 0.25 epochs.
Desired Behavior: Option to log every 10 number of steps.

Example of current logs:
{'loss': 2.8187, 'learning_rate': 2e-05, 'epoch': 0.25}
{'loss': 2.843, 'learning_rate': 1.959492736144978e-05, 'epoch': 0.5}
...

Use-case:
When training models, especially with larger datasets, epochs can take a significant amount of time. Having logs for every x number of steps would offer more frequent insights into the model's performance and learning rate adjustments.


Please help and suggest code to fix this issue.

Current command

torchrun --nproc_per_node=8 --master_port=20001 fastchat/train/train_mem.py \
  --model_name_or_path meta-llama/Llama-2-7b-hf  \
  --data_path data/dummy_conversation.json \
  --bf16 True \
  --output_dir output_vicuna_13b \
  --num_train_epochs 3 \
  --per_device_train_batch_size 4 \
  --per_device_eval_batch_size 32 \
  --gradient_accumulation_steps 4 \
  --evaluation_strategy "steps" \
  --eval_steps 1500 \
  --save_strategy "steps" \
  --save_steps 1500 \
  --save_total_limit 8 \
  --learning_rate 2e-5 \
  --weight_decay 0. \
  --warmup_ratio 0.04 \
  --lr_scheduler_type "cosine" \
  --logging_steps 1 \
  --fsdp "full_shard auto_wrap offload" \
  --fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer' \
  --tf32 True \
  --model_max_length 2048 \
  --gradient_checkpointing True \
  --lazy_preprocess False

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with fastchat/train/train_mem.py and trace the --logging_steps argument in the provided torchrun command. Confirm whether the existing training configuration supports step-based logging; done means logs are emitted at the requested interval, such as every 10 steps, rather than only at epoch intervals.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.