deepspeedai / deepspeedai/DeepSpeedExamples
Ignore index for mpu cross entropy
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
It seems to me that https://github.com/microsoft/DeepSpeedExamples/blob/fa1d1a71c48623db8a091d9cf636a5fe3b8f43c7/Megatron-LM/pretrain_bert.py#L221-L222 does not take ignore_index=-1 into account, while it does for bing-bert example: https://github.com/microsoft/DeepSpeedExamples/blob/a3cfec7c0f4aeaee2f0bb0f52b21dbf9637ac4f2/bing_bert/nvidia/modelingpreln.py#L1148
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read Megatron-LM/pretrain_bert.py at lines 221-222 and compare it with bing_bert/nvidia/modelingpreln.py at line 1148. Verify how the two cross-entropy paths handle ignore_index=-1; the work is done when the MPU path accounts for it consistently with the referenced BERT example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100