deepspeedai / deepspeedai/DeepSpeed

ZeRO stage 1 has different convergence compared to stage 0 and 2

Open
#757 2 comments 0 reactions 2 assignees View on GitHub

@awan-10 is already working on this.

Since Feb 16, 2021.

Dominant language
Python
Stars
43.1k
Forks
5k
Avg merge
4d 15h
Merged PRs (30d)
112

Description

When I train transformer architecture (encoder + decoder) with ZeRO stage 1, it shows different loss curve compared to stage 0 and stage 2.
On the other hand, the trainings with stage 0 and stage 2 look very similar.

transformer-zero

23: stage 0, 24: stage 1, 25: stage 2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.