NVIDIA / NVIDIA/Megatron-LM

[BUG] cross_entroy_loss_fusion with TE < 2.1 error

Open
#1,481 2 comments 0 reactions 0 assignees View on GitHub
bug community-request module: transformer engine
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

in language_module.py, when use cross_entropy_loss_fusion with TE < 2.1, te_parallel_cross_entropy will return None, and cause loss ERROR.

labels = labels.transpose(0, 1).contiguous()
if self.config.cross_entropy_loss_fusion:
if self.config.cross_entropy_fusion_impl == 'te':
if te_parallel_cross_entropy is not None:
labels = torch.as_strided(labels, labels.size(), (labels.size()[1], 1))
loss = te_parallel_cross_entropy(logits, labels)
else:
raise RuntimeError("Trying to use a TE block when it's not present.")
elif self.config.cross_entropy_fusion_impl == 'native':
loss = fused_vocab_parallel_cross_entropy(logits, labels)
else:
loss = tensor_parallel.vocab_parallel_cross_entropy(logits, labels

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.