enhancement bert training op clustering
- Dominant language
- C++
- Stars
- 932
- Forks
- 169
- PR merge metrics
- No merged PRs in 30d
Description
To utilize BladeDISC accelerating bert training, we need to cluster more op into disc, as the experiment, we found 244 `FusionGroups` in the first iteration with LTC, to reduce the clusters, we should:
- enhancement op coverage, #441 lists all ops that required, I just list some priority ones:
- native_layer_norm
- _softmax
- _log_softmax
- _log_softmax_backward_data
- _nll_loss_backward
- _nll_loss_forward
- tanh_backward
- native_dropout_backward
- gelu_backward
- enhancement shape propagation
- native_layer_norm
- native_dropout
- native_dropout_backward
[here](https://gist.github.com/Yancey1989/2ffbdea3c48baa0243e56d436eecfc98) list the 224 FusionGroups
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.