modelscope / modelscope/DiffSynth-Studio
Choice of `lora_target_modules`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
Hi DiffSynth team, many thanks for open-sourcing this great repo. I have a question regarding the lora_target_modules choice when finetuning the QWenImageEdit series. For example for img_mlp, why only lora finetuning the img_mlp.net.2 although there is also a linear layer in img_mlp.net.0.proj? Also why isn't the proj_out being finetuned? Any specific reason to keep those component frozen?
class QwenFeedForward(nn.Module):
def __init__(
self,
dim: int,
dim_out: Optional[int] = None,
dropout: float = 0.0,
):
super().__init__()
inner_dim = int(dim * 4)
self.net = nn.ModuleList([])
self.net.append(ApproximateGELU(dim, inner_dim))
self.net.append(nn.Dropout(dropout))
self.net.append(nn.Linear(inner_dim, dim_out))
def forward(self, hidden_states: torch.Tensor, *args, **kwargs) -> torch.Tensor:
for module in self.net:
hidden_states = module(hidden_states)
return hidden_states
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the QWenImageEdit LoRA configuration and the QwenFeedForward implementation shown in the issue. Trace how lora_target_modules selects layers, including img_mlp.net.2, img_mlp.net.0.proj, and proj_out. Done means the repository documents the rationale for the selected and frozen components.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100