linkedin / linkedin/Liger-Kernel
The accuracy is misaligned when using bf16
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 603
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 47
Description
### 🐛 Describe the bug
It is normal when using fp16, but there are some problems with bf16.
It seems that because swiglu uses tl.sigmoid(), it needs to be converted to fp32, but there is a loss of accuracy when converting to bf16.
This part of the code:
@triton.jit
def silu(x):
return x * tl.sigmoid(x)
### Reproduce
_No response_
### Versions
v0.4.2
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start from the provided Triton silu entry point and reproduce the reported difference between fp16 and bf16 using version v0.4.2. Compare the accuracy behavior around tl.sigmoid() and confirm the issue is resolved with the same comparison, since no reproduction case or test is provided.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100