deepspeedai / deepspeedai/DeepSpeed

[BUG]如何实现transformers、DeepSpeed单机多GPU训练、学习率使用cosine方法?

Open
#4,846 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug training
Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

Describe the bug
A clear and concise description of what the bug is.
训练过程中默认使用了deepspeed配置文件中的WarmupLR方式,通过搜索一些资料,若是希望实现cosine方法调整学习率,deepspeed配置文件删除scheduler即可,操作后训练报错:ValueError: You cannot create a DummyScheduler without specifying a scheduler in the config file.,定位至源码,报错原因就是配置文件中删除了scheduler部分,请问该如何实现呢?
transformer版本为4.34.0.
To Reproduce
Steps to reproduce the behavior:

  1. Go to '...'
  2. Click on '....'
  3. Scroll down to '....'
  4. See error

Expected behavior
A clear and concise description of what you expected to happen.

ds_report output
Please run ds_report to give us details about your setup.

Screenshots
If applicable, add screenshots to help explain your problem.

System info (please complete the following information):

  • OS: [e.g. Ubuntu 18.04]
  • GPU count and types [e.g. two machines with x8 A100s each]
  • Interconnects (if applicable) [e.g., two machines connected with 100 Gbps IB]
  • Python version
  • Any other relevant info about your setup

Launcher context
Are you launching your experiment with the deepspeed launcher, MPI, or something else?

Docker context
Are you using a specific docker image that you can share?

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the failure with Transformers 4.34.0, the reported DeepSpeed configuration, and single-machine multi-GPU training. Inspect the scheduler configuration path that raises the DummyScheduler error, then verify that cosine learning-rate training starts successfully and document the required configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.