[BUG] cross_entroy_loss_fusion with TE < 2.1 error
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
in language_module.py, when use cross_entropy_loss_fusion with TE < 2.1, te_parallel_cross_entropy will return None, and cause loss ERROR.
labels = labels.transpose(0, 1).contiguous()
if self.config.cross_entropy_loss_fusion:
if self.config.cross_entropy_fusion_impl == 'te':
if te_parallel_cross_entropy is not None:
labels = torch.as_strided(labels, labels.size(), (labels.size()[1], 1))
loss = te_parallel_cross_entropy(logits, labels)
else:
raise RuntimeError("Trying to use a TE block when it's not present.")
elif self.config.cross_entropy_fusion_impl == 'native':
loss = fused_vocab_parallel_cross_entropy(logits, labels)
else:
loss = tensor_parallel.vocab_parallel_cross_entropy(logits, labels
Contributor guide
Assessment
This issue has not been assessed yet.