huggingface / huggingface/peft

Silently slower delta-rule fallback when training a Qwen3.8-27B LoRA with examples/sft under FSDP on 8x B200

Open
#3,718 1 comment 0 reactions 1 assignee Claimed by @BenjaminBossan View on GitHub
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 12h
Merged PRs (30d)
59

Description

### System Info

peft 0.20.1.dev0 (main at f5ea6b00), transformers 5.16.1, accelerate 1.14.0, trl 1.12.0, torch 2.14.0+cu130, Python 3.11.16, Linux, one node with 8x NVIDIA B200, CUDA 13.0. No flash-attn, fla or hub kernels installed.

### Who can help?

_No response_

### Reproduction

I ran `examples/sft/train.py` as in `examples/sft/run_peft_fsdp.sh` (8-process FSDP, bf16) on Qwen/Qwen3.8-27B with LoRA r=8 and gradient accumulation 4, on 8x B200, with peft installed from main and no `fla` or flash-attn, and got 52.8 s per optimizer step. A profile showed the 48 linear-attention layers running transformers' pure-torch delta-rule fallback (`transformers/models/qwen3_5/modeling_qwen3_5.py:294-316`), which writes each chunk's result into a slice of the output, so the backward was mostly copy and select nodes. With the fallback reading chunk views instead of writing into slices, the same step took 41.7 s (from 53.0 s in that measurement run); a smaller second change, SHARD_GRAD_OP in the FSDP config, took it to 35.8 s.

I opened #3712 to record my experiment: setup, measurements, traces and some potential fixes. A different fix may also well suit the codebase.

### Expected behavior

Nothing in the logs said which delta-rule implementation was in use, so I only found it by profiling. One log line at model load naming the delta-rule path that was picked would have saved me the profile.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.