huggingface / huggingface/peft
Silently slower delta-rule fallback when training a Qwen3.8-27B LoRA with examples/sft under FSDP on 8x B200
- Dominant language
- Python
- Stars
- 21.7k
- Forks
- 2.5k
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 59
Description
### System Info
peft 0.20.1.dev0 (main at f5ea6b00), transformers 5.16.1, accelerate 1.14.0, trl 1.12.0, torch 2.14.0+cu130, Python 3.11.16, Linux, one node with 8x NVIDIA B200, CUDA 13.0. No flash-attn, fla or hub kernels installed.
### Who can help?
_No response_
### Reproduction
I ran `examples/sft/train.py` as in `examples/sft/run_peft_fsdp.sh` (8-process FSDP, bf16) on Qwen/Qwen3.8-27B with LoRA r=8 and gradient accumulation 4, on 8x B200, with peft installed from main and no `fla` or flash-attn, and got 52.8 s per optimizer step. A profile showed the 48 linear-attention layers running transformers' pure-torch delta-rule fallback (`transformers/models/qwen3_5/modeling_qwen3_5.py:294-316`), which writes each chunk's result into a slice of the output, so the backward was mostly copy and select nodes. With the fallback reading chunk views instead of writing into slices, the same step took 41.7 s (from 53.0 s in that measurement run); a smaller second change, SHARD_GRAD_OP in the FSDP config, took it to 35.8 s.
I opened #3712 to record my experiment: setup, measurements, traces and some potential fixes. A different fix may also well suit the codebase.
### Expected behavior
Nothing in the logs said which delta-rule implementation was in use, so I only found it by profiling. One log line at model load naming the delta-rule path that was picked would have saved me the profile.
Contributor guide
Assessment
This issue has not been assessed yet.