modelscope / modelscope/DiffSynth-Studio
按照OminiControl的实现思路训练lora,似乎无法调起底模应有的能力?
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.1k
- Forks
- 1.3k
- Avg merge
- 13h 12m
- Merged PRs (30d)
- 45
Description
按照OminiControl的实现思路,通过多条件拼接的方法,在位置对齐任务上出现纹理较差的问题,在非位置对齐的ID保持上无法有效的控制ID的生成(ID保持不像),尽管在单张图上通过了拟合的实验。并且以上数据集在flux上均通过了验证,并且表现较好。
我做了如下的一些实验和思考
1、完全复制的任务,条件图和目标图是同一张,生成效果没有问题,这个证明了整个的采样过程是没有问题的,这个任务在自己写的基于diffusers的代码上也通过验证
2、风格加IP保持的任务,类似OmniConsistency的任务,发现模型对于风格是最先学习到的,而ip基本上没有学习到。后面在做cfg时,负向丢弃图像条件,ip会有一点点像。
3、完全对齐的消除任务,出现的纹理较差不自然,对于简单的空白背景会好一些
最开始我用的diffusers的框架去改写的,一直怀疑采样过程没有对齐还有位置编码,实际上这些应该不影响大致的效果,比如ID信息无法保留的问题。后面采用DiffSynth-Studio训练相同数据,结论基本差不多。我怀疑过cfg推理时的问题,因为训练的损失真的很低,比flux低很多。
官方是否进行过这方面的验证,按道理说qwen-image应该在这方面会更加强,还是说这样的方式并不适合qwen-image
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by reproducing the reported OminiControl-style LoRA training cases for position alignment and ID preservation in DiffSynth-Studio, then compare them with the stated diffusers and FLUX results. Done would require an agreed diagnosis or documented validation of whether the approach works for qwen-image.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100