huggingface / huggingface/peft
target_modules can match a subset of layers with no warning
- Dominant language
- Python
- Stars
- 21.7k
- Forks
- 2.5k
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 59
Description
### Feature request
`target_modules` is resolved by module name suffix, and `ValueError` is raised only
when the whole list matched nothing. If a name matches on some layers and not on
others, the adapter is built quietly. The only visible sign is a trainable parameter
count, and people rarely hand verify that.
I would like the coverage to be visible. `get_model_status()` and
`get_layer_status()` already exist and look like the natural place: for each entry in
`target_modules`, how many modules it matched and on which layer indices. Then a user
can assert on it in their own code.
A warning would work too, something along the lines of `target 'v_proj' matched 25
modules on layers [0-4, 6-10, ...], other targets matched 30`, emitted only when
coverage is uneven across targets. I do not know how noisy that would be across the
model zoo, so the reporting version seems like the safer ask.
### Motivation
This is not an edge case any more. `google/gemma-4-26B-A4B` has 30 text layers, and
five of them (5, 11, 17, 23, 29) are global attention layers where the model reuses
the key projection as the value projection. `Gemma4TextConfig` carries
`attention_k_eq_v: true`, and `Gemma4TextAttention.__init__` reads:
self.use_alternative_attention = config.attention_k_eq_v and not self.is_sliding
self.v_proj = nn.Linear(...) if not self.use_alternative_attention else None
The checkpoint has no `self_attn.v_proj.weight` for those five layers. So
`target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]` gives `v_proj` on 25 layers
and the other three on 30. The adapter is asymmetric across depth and nothing says so.
Repro:
```python
from collections import Counter
from transformers import AutoModelForCausalLM
from peft import LoraConfig, get_peft_model
model = AutoModelForCausalLM.from_pretrained(
"google/gemma-4-26B-A4B", torch_dtype="auto", device_map="auto")
model = get_peft_model(model, LoraConfig(
task_type="CAUSAL_LM", r=32, lora_alpha=64,
target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]))
print(Counter(
n.split(".lora_")[0].rsplit(".", 1)[-1]
for n, p in model.named_parameters()
if p.requires_grad and "lora_A" in n
))
# v_proj: 25, q_proj: 30, k_proj: 30, o_proj: 30
```
Real configs hit this. Published Gemma 4 adapters use regexes like
`(mlp|self_attn)\.(up|down|gate|q|k|v|o)_proj` that treat `v` uniformly across depth
(the axolotl config in the `zerofata/G4-MeroMero-26B-A4B` model card is one public
example). Several third party Gemma 4 fine-tuning guides recommend the seven name
list with no caveat about layer coverage. `target_modules="all-linear"` is fine,
because it enumerates what exists rather than what you named.
The consequence is worse than a smaller adapter, at least on this architecture. On
those five layers `k_proj` is the value matrix, so an adapter you believe is queries
and keys only is editing the value path, and an adapter you believe is values and
output only cannot reach values there at all. Anyone comparing QK against VO
adaptation on Gemma 3/4, Qwen3 or OLMo is comparing something other than what they
configured.
None of this is PEFT's fault. But PEFT is the layer that knows what matched, and it
is the only place a fix does not have to be repeated in every model card.
### Contribution
Happy to open a PR against `get_model_status()` / `get_layer_status()` if the
direction is acceptable. Details and the forward pass walkthrough:
https://huggingface.co/SubMaroon/gemma4-lora-traps
Contributor guide
Research direction
Start by reading the existing get_model_status() and get_layer_status() entry points, then reproduce the Gemma 4 target_modules example to inspect the current matching information. Done means each target reports its matched module count and layer indices, making uneven coverage visible to callers without relying on parameter counts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100