NVIDIA / NVIDIA/Megatron-LM

[QUESTION] How to quantize weight of te.linear in megatron?

Open
#1,789 3 comments 0 reactions 0 assignees View on GitHub
community-request enhancement module: transformer engine waiting-on-customer
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

I want to fake quantize weight of te.linear layer in megatron. If I use pytorch, the code is simple like this:
```
def forward(self, x_quant):
w_quant = fake_quant(self.weight)
y = torch.nn.functional.linear(x_quant, w_quant)
return y
```
However, when applying this method to te.linear, it will raise an error due to `w_quant` lacking `main.grad`. How can I solve this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.