modelscope / modelscope/DiffSynth-Studio

Choice of `lora_target_modules`

Open
#1,203 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

Hi DiffSynth team, many thanks for open-sourcing this great repo. I have a question regarding the lora_target_modules choice when finetuning the QWenImageEdit series. For example for img_mlp, why only lora finetuning the img_mlp.net.2 although there is also a linear layer in img_mlp.net.0.proj? Also why isn't the proj_out being finetuned? Any specific reason to keep those component frozen?

class QwenFeedForward(nn.Module):
    def __init__(
        self,
        dim: int,
        dim_out: Optional[int] = None,
        dropout: float = 0.0,
    ):
        super().__init__()
        inner_dim = int(dim * 4)
        self.net = nn.ModuleList([])
        self.net.append(ApproximateGELU(dim, inner_dim))
        self.net.append(nn.Dropout(dropout))
        self.net.append(nn.Linear(inner_dim, dim_out))

    def forward(self, hidden_states: torch.Tensor, *args, **kwargs) -> torch.Tensor:
        for module in self.net:
            hidden_states = module(hidden_states)
        return hidden_states

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the QWenImageEdit LoRA configuration and the QwenFeedForward implementation shown in the issue. Trace how lora_target_modules selects layers, including img_mlp.net.2, img_mlp.net.0.proj, and proj_out. Done means the repository documents the rationale for the selected and frozen components.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.