linkedin / linkedin/Liger-Kernel

[B300/SM103] Compiled Liger SwiGLU is ~3.1x slower than torch.compile for a Qwen3-shaped BF16 tensor

Open
#1,330 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.6k
Forks
603
Avg merge
1d 20h
Merged PRs (30d)
47

Description

## 🐛 Describe the bug

On an NVIDIA B300 (SM103), the Blackwell Liger SwiGLU path is substantially
slower than the equivalent `torch.compile` expression for a Qwen3-shaped BF16
tensor.

With shape `(3072, 12288)`, including forward and backward:

| implementation | mean step time |
|---|---:|
| `torch.compile(lambda a, b: F.silu(a) * b)` | 2.169 ms |
| `torch.compile(lambda a, b: LigerSiLUMulFunction.apply(a, b))` | 6.831 ms |

The Liger path takes approximately 3.15x as long. The result reproduces on the
current Liger main (`91ae44ae659ebaf40d4a851777f4f02515e9ba65`), so it does
not appear to be specific to the 0.8.1 release.

This was initially observed in Qwen3-8B DFlash training. At sequence length
8192 and `max_anchors=3072`, enabling Liger increased the end-to-end training
step from 176.20 ms to 223.37 ms (+26.77%) and reduced throughput from 46.54k
to 36.70k tokens/s (-21.14%). Three alternating runs were used for each mode,
with 10 warmup and 50 measured steps per run.

The outputs and gradients remain numerically consistent at BF16 tolerances, so
this report is about performance rather than correctness.

## Reproduce

```python
import json
import statistics

import torch
import torch.nn.functional as F
from liger_kernel.ops.swiglu import LigerSiLUMulFunction

ROWS = 3072
COLS = 12288
WARMUP = 10
STEPS = 50

def benchmark(function):
compiled = torch.compile(function)
torch.manual_seed(42)
a = torch.randn(
(ROWS, COLS), device="cuda", dtype=torch.bfloat16, requires_grad=True
)
b = torch.randn(
(ROWS, COLS), device="cuda", dtype=torch.bfloat16, requires_grad=True
)
torch.manual_seed(43)
gradient = torch.randn_like(a)

def step():
a.grad = None
b.grad = None
compiled(a, b).backward(gradient)

for _ in range(WARMUP):
step()
torch.cuda.synchronize()

samples = []
for _ in range(STEPS):
start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)
start.record()
step()
end.record()
end.synchronize()
samples.append(start.elapsed_time(end))

return {
"mean_ms": statistics.mean(samples),
"median_ms": statistics.median(samples),
"stdev_ms": statistics.stdev(samples),
}

native = benchmark(lambda a, b: F.silu(a) * b)
liger = benchmark(lambda a, b: LigerSiLUMulFunction.apply(a, b))
print(
json.dumps(
{
"native": native,
"liger": liger,
"liger_over_native": liger["mean_ms"] / native["mean_ms"],
},
indent=2,
)
)
```

Profiling attributes most of the difference to the tiled Liger activation
kernels rather than the surrounding GEMMs. Disabling the Blackwell tiled path
made the isolated Liger result slower still (about 7.88 ms), so tiling helps
relative to Liger's one-row path but does not close the gap with the compiled
native expression.

## Versions

- Liger Kernel: main at `91ae44ae659ebaf40d4a851777f4f02515e9ba65`
- GPU: NVIDIA B300 SXM6 AC
- Compute capability: 10.3 (SM103)
- PyTorch: `2.11.0+cu130`
- CUDA runtime: `13.0`
- OS: Linux 6.8.0, x86_64
- dtype: BF16

Contributor guide

Open the contributing guide

Research direction

Run the provided benchmark for LigerSiLUMulFunction against the torch.compile SiLU expression on the specified BF16 shape and environment. Profile the tiled Liger activation kernels, comparing them with the one-row path and native compiled expression. Done means the performance gap is reduced while outputs and gradients remain numerically consistent.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.