NVIDIA-NeMo / NVIDIA-NeMo/Automodel
Expert LoRA in Transformers v5
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 141
Description
With the new fused implementations (e.g., https://github.com/huggingface/transformers/blob/main/src/transformers/models/mixtral/modeling_mixtral.py#L71-L72) we no longer have an nn.Linear to match against anymore when we want to match with match_all_linear=True. The relevant parameters are registered via nn.Parameter. The recommended method is to pass in target_modules for the specific layer as well as rank_pattern for the per expert rank sizing. Automodel needs to provide support for this.
Note: as a result, we are currently not providing expert LoRA support if it's a HF native MoE model + fused experts implementation.
Additional reading: https://huggingface.co/docs/peft/main/en/developer_guides/lora#moe-expert-parameters-and-vllm
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Transformers' src/transformers/models/mixtral/modeling_mixtral.py at the fused expert implementation, then trace Automodel's LoRA/PEFT integration. Check how target_modules and rank_pattern are passed for expert parameters, and verify that native Hugging Face MoE models with fused experts receive expert LoRA support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100