linkedin / linkedin/Liger-Kernel
fused_linear_cross_entropy caused bad performance
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.6k
- Forks
- 603
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 47
Description
### 🐛 Describe the bug
Hello, i'm finetuning Qwen-2.5-VL-3B with trl. When i turn on the liger kernel, it could lead to poor performance while the performance is normal without liger kernel. I later discovered that it was `fused_linear_cross_entropy` that caused the performance difference. However, their loss looks almost the same, and the problem is that this option is crucial for saving GPU memory.
Here is my training args
deepspeed --master_port 25410 src/open_r1/sft.py \
--model_name_or_path .cache/Qwen2.5-VL-3B-Instruct \
--dataset_name data/instructions_dynamic_combine_action.jsonl \
--deepspeed scripts/zero3.json \
--learning_rate 2.0e-5 \
--num_train_epochs 1 \
--packing \
--max_seq_length 64800 \
--per_device_train_batch_size 4 \
--gradient_accumulation_steps 4 \
--dataloader_num_workers 4 \
--gradient_checkpointing True \
--torch_dtype bfloat16 \
--bf16 \
--logging_steps 5 \
--eval_strategy no \
--eval_steps 100 \
--save_strategy no \
--save_steps 6000 \
--attn_implementation flash_attention_2 \
--output_dir result/Qwen2.5-VL_3B_sft_r2r_dynamic_test_fix_liger \
--report_to tensorboard
### Reproduce
_No response_
### Versions
Environment Report:
-------------------
Operating System: Linux-6.8.0-54-generic-x86_64-with-glibc2.39
Python version: 3.10.16
Liger Kernel version: 0.5.10
PyTorch version: 2.7.1+cu126
CUDA version: 12.6
HIP(ROCm) version: Not available
Triton version: 3.3.1
Transformers version: 4.50.3
Trl version: 0.16.0
XPU version: XPU Not Available
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the fused_linear_cross_entropy entry point and the training command in src/open_r1/sft.py, using scripts/zero3.json and the reported environment as context. Compare training with and without the option, then establish a reproducible performance difference and identify the affected behavior; no dedicated test or reproduction is provided in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100