modelscope / modelscope/DiffSynth-Studio

关于find_unused_parameters的作用

Open
#912 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

Hi guys, 我想请教一下关于find_unused_parameters的作用

在qwen-image-edit的lora训练中,如果不设置find_unused_parameters的话会出现以下报错:

Parameters which did not receive grad for rank 0:

  • pipe.dit.transformer_blocks.59.txt_mlp.net.2.lora_B.default.weight
  • pipe.dit.transformer_blocks.59.txt_mlp.net.2.lora_A.default.weight
  • pipe.dit.transformer_blocks.59.attn.to_add_out.lora_B.default.weight
  • pipe.dit.transformer_blocks.59.attn.to_add_out.lora_A.default.weight

请问一下:
这个txt_mlp.net.2attn.to_add_outlora_target_modules中又指定,但是为什么在DDP中又需要设置为unused呢?这个会影响LoRA training的效果吗?以及find_unused_parameters的作用是什么?谢谢!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the qwen-image-edit LoRA training configuration and the DDP setting for find_unused_parameters. Trace why txt_mlp.net.2 and attn.to_add_out are listed in lora_target_modules but do not receive gradients, then explain whether this changes training behavior and what the setting does. No file or test is named in the issue; done means providing a clear, evidence-based explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
distributed-systems, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.