alibaba / alibaba/BladeDISC

enhancement bert training op clustering

Open
#473 0 comments 0 reactions 1 assignee Claimed by @Yancey0623 View on GitHub
Dominant language
C++
Stars
932
Forks
169
PR merge metrics
No merged PRs in 30d

Description

To utilize BladeDISC accelerating bert training, we need to cluster more op into disc, as the experiment, we found 244 `FusionGroups` in the first iteration with LTC, to reduce the clusters, we should:

- enhancement op coverage, #441 lists all ops that required, I just list some priority ones:
- native_layer_norm
- _softmax
- _log_softmax
- _log_softmax_backward_data
- _nll_loss_backward
- _nll_loss_forward
- tanh_backward
- native_dropout_backward
- gelu_backward
- enhancement shape propagation
- native_layer_norm
- native_dropout
- native_dropout_backward

[here](https://gist.github.com/Yancey1989/2ffbdea3c48baa0243e56d436eecfc98) list the 224 FusionGroups

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.