modelscope / modelscope/DiffSynth-Studio
关于如何修改模型结构的问题
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
您好，感谢您的工作，我有关于模型结构修改的两个问题：
- 在原始Wan2.1-Fun-V1.1-1.3B-Control模型中，模型输入为reference_image（参考图像）, control_video（控制视频），video（视频真值），现在我想将第一帧图像到视频生成模型和原模型结合起来，那么我在extra_inputs中还多添加了一个input_images，这样子在pipeline中会自动调用WanVideoUnit_ImageEmbedderVAE以更新y且不会改变模型结构，这样子是否能够训练一个新的模型
- 我对wan_video_dit模型作出结构上的修改，比如多增加了一个动作输入，是否在self.keys_hash_with_shape_dict中添加新模型检查点的hashkey就可以完成新模型的注册和使用
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the pipeline's handling of extra_inputs and WanVideoUnit_ImageEmbedderVAE, then inspect wan_video_dit and self.keys_hash_with_shape_dict. Determine how the proposed input and architecture changes affect training and checkpoint registration. Done would require a documented maintainer decision or a reproducible implementation plan.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100