linkedin / linkedin/Liger-Kernel

Compatibility issue between liger-kernel and zero3: The size of tensor a (0) must match the size of tensor b (4096) at non-singleton dimension 1

Open
#1,048 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
6.6k
Forks
603
Avg merge
1d 20h
Merged PRs (30d)
47

Description

### 🐛 Describe the bug

When I was training qwen3-8b using liger-kernel + zero3, an error occurred during backpropagation:
`The size of tensor a (0) must match the size of tensor b (4096) at non-singleton dimension 1`

After debugging, I found that changing the `stage3_param_persistence_threshold` parameter in the zero3 config from `auto` to `1e10` can solve this problem. Alternatively, changing zero3 to zero2 also works.

I want to ask, why is this? I might not have this problem when training gemma3. Can anyone help explain where the compatibility issue between liger-kernel and deepspeed is when training Qwen3?

Hope your answer, thanks!

### Reproduce

_No response_

### Versions
Operating System: Linux-5.15.0-126-generic-x86_64-with-glibc2.39
Python version: 3.12.12
Liger Kernel version: 0.5.10
PyTorch version: 2.7.1+cu126
CUDA version: 12.6
HIP(ROCm) version: Not available
Triton version: 3.3.1
Transformers version: 4.51.3
XPU version: XPU Not Available

Contributor guide

Open the contributing guide

Research direction

No source file, test, or reproduction steps are named. Start by reproducing Qwen3-8B training with Liger Kernel and DeepSpeed ZeRO-3 using the reported versions, then compare `stage3_param_persistence_threshold: auto` with `1e10` and ZeRO-2. Done means identifying the compatibility boundary and documenting or testing a corrective change.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.