huggingface / huggingface/peft

Transformers v5 MoE conversion incorrectly rewrites fully qualified shared-expert LoRA targets

Open
#3,711 5 comments 0 reactions 1 assignee Claimed by @BenjaminBossan View on GitHub
Dominant language
Python
Stars
21.7k
Forks
2.5k
Avg merge
4d 16h
Merged PRs (30d)
60

Description

### System Info

### Environment

- Model: `amd/Instella-MoE-16B-A3B-Think`
- PEFT: 0.20.0
- Transformers: 5.16.1
- `trust_remote_code=True`

### Who can help?

_No response_

### Reproduction

### Problem

Instella contains two different representations of MoE projections:

1. Routed experts use fused 3D parameters, for example:

`model.layers.12.mlp.experts.down_proj`

2. Shared experts are regular MLP modules containing `nn.Linear` layers, for example:

`model.layers.12.mlp.shared_experts.down_proj`

A mixed LoRA configuration therefore legitimately uses both `target_parameters` and `target_modules`:

```python
config = LoraConfig(
r=8,
target_modules=[
"model.layers.12.mlp.shared_experts.down_proj",
],
target_parameters=[
"model.layers.12.mlp.experts.down_proj",
],
)

peft_model = get_peft_model(model, config)
```

Expected behavior:
- `model.layers.12.mlp.shared_experts.down_proj` is retained as a module target.
- `model.layers.12.mlp.experts.down_proj` is retained as a parameter target.
- PEFT installs a regular LoRA layer on the shared expert and a parameter wrapper on the routed expert bank.

Actual behavior:
```
config.target_modules == set()
config.target_parameters == {"down_proj"}
```

Only routed-expert parameter wrappers are created. The shared-expert `nn.Linear` module is not adapted.
The fully qualified, layer-specific target is also reduced to the generic name `down_proj`, potentially broadening it to other matching expert parameters.

### Cause
Instella reports `model_type="deepseek_v3"`. When Transformers v5 conversion is enabled, PEFT applies the `deepseek_v3 -> qwen2_moe` checkpoint conversion mapping.
The MoE conversion logic treats any target module equal to, or ending in `down_proj` as an old routed-expert target. It does not check whether the path refers to `.experts.` or `.shared_experts.`.

Consequently, this ordinary linear module:
```
model.layers.12.mlp.shared_experts.down_proj
```
is mistaken for a fused routed-expert parameter. It is removed from `target_modules`, and only its leaf name, `down_proj`, is added to target_parameters.

This conversion is also applied to newly created `LoraConfig` objects passed to `get_peft_model`; it is not limited to loading legacy adapters.

Relevant model files:
- [https://huggingface.co/amd/Instella-MoE-16B-A3B-Think/blob/main/config.json](https://huggingface.co/amd/Instella-MoE-16B-A3B-Think/blob/main/config.json)
- [https://huggingface.co/amd/Instella-MoE-16B-A3B-Think/blob/main/modeling_instella_moe.py](https://huggingface.co/amd/Instella-MoE-16B-A3B-Think/blob/main/modeling_instella_moe.py)

Relevant PEFT conversion logic:
- [https://github.com/huggingface/peft/blob/main/src/peft/utils/transformers_weight_conversion.py](https://github.com/huggingface/peft/blob/main/src/peft/utils/transformers_weight_conversion.py)

### Reproduction Code:
```
from types import SimpleNamespace

import torch
from peft import LoraConfig
from peft.utils.transformers_weight_conversion import (
convert_peft_config_for_transformers,
)

# Instella declares model_type="deepseek_v3".
model = torch.nn.Module()
model.config = SimpleNamespace(model_type="deepseek_v3")

config = LoraConfig(
r=8,
target_modules=[
"model.layers.12.mlp.shared_experts.down_proj",
],
target_parameters=[
"model.layers.12.mlp.experts.down_proj",
],
)

print("Before:")
print(" target_modules: ", config.target_modules)
print(" target_parameters:", config.target_parameters)

convert_peft_config_for_transformers(config, model, conversions=[])

print("After:")
print(" target_modules: ", config.target_modules)
print(" target_parameters:", config.target_parameters)
```

### Expected behavior

The conversion should distinguish routed expert paths from ordinary/shared MLP paths.
Possible approaches:
- Apply the MoE module-to-parameter conversion only to targets structurally inside a routed `.experts.` container.
- Preserve fully qualified paths rather than reducing them to the final component.
- Convert only known legacy routed-expert path forms, while leaving `.shared_experts.*` targets untouched.
- Inspect whether the resolved target is an `nn.Module` or a direct fused parameter before changing `target_modules` into `target_parameters`.

Bare targets such as "down_proj" may need the existing broad behavior for backward compatibility, but a fully qualified `.shared_experts.down_proj` target should not be interpreted as a routed-expert bank.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.