deepspeedai / deepspeedai/DeepSpeedExamples
Advice on the experiment
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
Hi! In my experiment I have to use the optimizer SGD and my model can't adapt to the float16/bfloat16 mode cause its accuracy will drop rapidly. I have tried as the tutorial says, and added additional setting "zero_force_ds_cpu_optimizer": false,
"zero_allow_untested_optimizer": true
But my GPU memory seens to have no changes. Could you please provide some advice on this optimization problem? Thanks!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked Transformers DeepSpeed ZeRO-2 tutorial and reproduce the reported setup using SGD with float16 or bfloat16. Compare GPU memory with and without the two added settings, then document why memory does or does not change and what configuration is required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100