linkedin / linkedin/Liger-Kernel
Compatibility issue between liger-kernel and zero3: The size of tensor a (0) must match the size of tensor b (4096) at non-singleton dimension 1
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 603
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 47
Description
### 🐛 Describe the bug
When I was training qwen3-8b using liger-kernel + zero3, an error occurred during backpropagation:
`The size of tensor a (0) must match the size of tensor b (4096) at non-singleton dimension 1`
After debugging, I found that changing the `stage3_param_persistence_threshold` parameter in the zero3 config from `auto` to `1e10` can solve this problem. Alternatively, changing zero3 to zero2 also works.
I want to ask, why is this? I might not have this problem when training gemma3. Can anyone help explain where the compatibility issue between liger-kernel and deepspeed is when training Qwen3?
Hope your answer, thanks!
### Reproduce
_No response_
### Versions
Operating System: Linux-5.15.0-126-generic-x86_64-with-glibc2.39
Python version: 3.12.12
Liger Kernel version: 0.5.10
PyTorch version: 2.7.1+cu126
CUDA version: 12.6
HIP(ROCm) version: Not available
Triton version: 3.3.1
Transformers version: 4.51.3
XPU version: XPU Not Available
Contributor guide
Research direction
No source file, test, or reproduction steps are named. Start by reproducing Qwen3-8B training with Liger Kernel and DeepSpeed ZeRO-3 using the reported versions, then compare `stage3_param_persistence_threshold: auto` with `1e10` and ZeRO-2. Done means identifying the compatibility boundary and documenting or testing a corrective change.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100