bigscience-workshop / bigscience-workshop/Megatron-DeepSpeed
"Mask is silently ignored due to the use of a custom kernel" with pretrain_gpt_single_node.sh
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 226
- PR merge metrics
- No merged PRs in 30d
Description
## Update
The issue turned out to be DeepSpeed usages of `pretrain_gpt_single_node.sh`.
I will make a pull request soon.
## Original Report
Please let me know what details I shall provide, thank you!
Python 3.7.12
pt=1.11.0+cu113, cuda=11.3
Using `pretrain_gpt_single_node.sh` with the instructions on README:
```
[...]/megatron/model/fused_softmax.py", line 191, in forward_fused_softmax
assert mask is None, "Mask is silently ignored due to the use of a custom kernel"
```
Also tried `--no-masked-softmax-fusion` but no differences except the line.
```
[...]/megatron/model/fused_softmax.py", line 218, in forward_torch_softmax
assert mask is None
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.