bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed

How can we access to the gradients while the model is training?

Open
#377 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
226
PR merge metrics
No merged PRs in 30d

Description

Does deepspeed offer an API to access the gradients while training for any of the stages (1-2-3). When I try to access the gradients, I only get None types. I'm mostly interested in the stage 3. If there is no such API, is there any chance to delve into the deepspeed code to access them somehow, I would also be glad if someone just points me to the right direction in the source code for this. I'm not that concerned about any slowdowns in the code as long as I have access to the gradients.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.