huggingface / huggingface/peft

target_modules can match a subset of layers with no warning

Open
#3,658 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 12h
Merged PRs (30d)
59

Description

### Feature request

`target_modules` is resolved by module name suffix, and `ValueError` is raised only
when the whole list matched nothing. If a name matches on some layers and not on
others, the adapter is built quietly. The only visible sign is a trainable parameter
count, and people rarely hand verify that.

I would like the coverage to be visible. `get_model_status()` and
`get_layer_status()` already exist and look like the natural place: for each entry in
`target_modules`, how many modules it matched and on which layer indices. Then a user
can assert on it in their own code.

A warning would work too, something along the lines of `target 'v_proj' matched 25
modules on layers [0-4, 6-10, ...], other targets matched 30`, emitted only when
coverage is uneven across targets. I do not know how noisy that would be across the
model zoo, so the reporting version seems like the safer ask.

### Motivation

This is not an edge case any more. `google/gemma-4-26B-A4B` has 30 text layers, and
five of them (5, 11, 17, 23, 29) are global attention layers where the model reuses
the key projection as the value projection. `Gemma4TextConfig` carries
`attention_k_eq_v: true`, and `Gemma4TextAttention.__init__` reads:

self.use_alternative_attention = config.attention_k_eq_v and not self.is_sliding
self.v_proj = nn.Linear(...) if not self.use_alternative_attention else None

The checkpoint has no `self_attn.v_proj.weight` for those five layers. So
`target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]` gives `v_proj` on 25 layers
and the other three on 30. The adapter is asymmetric across depth and nothing says so.

Repro:

```python
from collections import Counter
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model

model = AutoModelForCausalLM.from_pretrained(
"google/gemma-4-26B-A4B", torch_dtype="auto", device_map="auto")
model = get_peft_model(model, LoraConfig(
task_type="CAUSAL_LM", r=32, lora_alpha=64,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]))

print(Counter(
n.split(".lora_")[0].rsplit(".", 1)[-1]
for n, p in model.named_parameters()
if p.requires_grad and "lora_A" in n
))
# v_proj: 25, q_proj: 30, k_proj: 30, o_proj: 30
```

Real configs hit this. Published Gemma 4 adapters use regexes like
`(mlp|self_attn)\.(up|down|gate|q|k|v|o)_proj` that treat `v` uniformly across depth
(the axolotl config in the `zerofata/G4-MeroMero-26B-A4B` model card is one public
example). Several third party Gemma 4 fine-tuning guides recommend the seven name
list with no caveat about layer coverage. `target_modules="all-linear"` is fine,
because it enumerates what exists rather than what you named.

The consequence is worse than a smaller adapter, at least on this architecture. On
those five layers `k_proj` is the value matrix, so an adapter you believe is queries
and keys only is editing the value path, and an adapter you believe is values and
output only cannot reach values there at all. Anyone comparing QK against VO
adaptation on Gemma 3/4, Qwen3 or OLMo is comparing something other than what they
configured.

None of this is PEFT's fault. But PEFT is the layer that knows what matched, and it
is the only place a fix does not have to be repeated in every model card.

### Contribution

Happy to open a PR against `get_model_status()` / `get_layer_status()` if the
direction is acceptable. Details and the forward pass walkthrough:
https://huggingface.co/SubMaroon/gemma4-lora-traps

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing get_model_status() and get_layer_status() entry points, then reproduce the Gemma 4 target_modules example to inspect the current matching information. Done means each target reports its matched module count and layer indices, making uneven coverage visible to callers without relying on parameter counts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.