[QUESTION] why WrappedTorchLayerNorm sequence parallel not supported by torch LayerNorm？
Open
community-request
question
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
@cuichenx just like TENorm add
# Set flag for sequence parallelism (custom Megatron-LM integration)
if getattr(self, "sequence_parallel", None) is not None:
self.weight.sequence_parallel = self.sequence_parallel
self.bias.sequence_parallel = self.sequence_parallel to WrappedTorchLayerNorm class can support sequence parallel ？
Contributor guide
Assessment
This issue has not been assessed yet.