deepspeedai / deepspeedai/DeepSpeedExamples
'gamma', 'theta' not found in progressive layer drop
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
Hi! Thank you guys for the tool and the example.
I've been trying to reproduce 'progressive layer dropping' on Roberta and other pretrain methods, but I didn't found where gamma and theta which stated in deepspeed_bsz4k_progressive_layer_drop_config_seq128.json are used in the project.
For 'theta', I found code in nvidia/modelingpreln_layerdrop.py line 1160 theta = kwargs.get('pld_theta', 1.0), but it's 'pld_theta' rather than 'theta'.
For 'gamma', which should be used for drop rate scheduling, I didn't find anywhere using it.
Please kindly inform me if I miss anything, thank you very much.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with deepspeed_bsz4k_progressive_layer_drop_config_seq128.json and nvidia/modelingpreln_layerdrop.py around line 1160. Trace how the progressive layer-drop configuration is loaded and how pld_theta and the drop-rate schedule are consumed. Done means the roles of gamma and theta are accounted for in the implementation or documented as unsupported, with focused verification for the configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100