modelscope / modelscope/DiffSynth-Studio

按照OminiControl的实现思路训练lora,似乎无法调起底模应有的能力?

Open
#777 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

按照OminiControl的实现思路,通过多条件拼接的方法,在位置对齐任务上出现纹理较差的问题,在非位置对齐的ID保持上无法有效的控制ID的生成(ID保持不像),尽管在单张图上通过了拟合的实验。并且以上数据集在flux上均通过了验证,并且表现较好。
我做了如下的一些实验和思考
1、完全复制的任务,条件图和目标图是同一张,生成效果没有问题,这个证明了整个的采样过程是没有问题的,这个任务在自己写的基于diffusers的代码上也通过验证
2、风格加IP保持的任务,类似OmniConsistency的任务,发现模型对于风格是最先学习到的,而ip基本上没有学习到。后面在做cfg时,负向丢弃图像条件,ip会有一点点像。
3、完全对齐的消除任务,出现的纹理较差不自然,对于简单的空白背景会好一些

最开始我用的diffusers的框架去改写的,一直怀疑采样过程没有对齐还有位置编码,实际上这些应该不影响大致的效果,比如ID信息无法保留的问题。后面采用DiffSynth-Studio训练相同数据,结论基本差不多。我怀疑过cfg推理时的问题,因为训练的损失真的很低,比flux低很多。

官方是否进行过这方面的验证,按道理说qwen-image应该在这方面会更加强,还是说这样的方式并不适合qwen-image

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are named. Start by reproducing the reported OminiControl-style LoRA training cases for position alignment and ID preservation in DiffSynth-Studio, then compare them with the stated diffusers and FLUX results. Done would require an agreed diagnosis or documented validation of whether the approach works for qwen-image.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.