modelscope / modelscope/DiffSynth-Studio

ckpt lora 和comfyui的兼容性

Open
#594 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

我使用框架训练了一个 ckpt格式的lora模型 使用代码进行测试的话
`from diffsynth import ModelManager, FluxImagePipeline
import torch

model_manager = ModelManager(torch_dtype=torch.bfloat16, device="cuda",
file_path_list=[
"models/FLUX/FLUX.1-dev/text_encoder/model.safetensors",
"models/FLUX/FLUX.1-dev/text_encoder_2",
"models/FLUX/FLUX.1-dev/ae.safetensors",
"models/FLUX/FLUX.1-dev/flux1-dev.safetensors"
])
model_manager.load_lora(r"loras/lightning_logs/version_4/checkpoints/epoch=19-step=2000.ckpt", lora_alpha=1.0)
pipe = FluxImagePipeline.from_model_manager(model_manager)

torch.manual_seed(0)
image = pipe(
prompt="A traditional chinese painting depicting a group of men in traditional clothing engaged in a conversation. the scene is set in a traditional indoor setting with a tiled floor and a wooden fence in the background. on the left side of the image, a man with a mustache and a serious expression is standing and holding a plate in his hand, wearing a pink robe and a blue hat. he appears to be in his mid-twenties, with a slim body and a prominent facial expression. he is facing the viewer and pointing at another man who is seated in front of him, who is wearing a blue and gold outfit with intricate patterns and a long beard. to his right, another man is sitting on the floor, also wearing a similar outfit with a similar pattern and has a similar facial expression and is facing another man. to the left of the man in the pink robe, there is another man sitting on a bench, and to the right, there are six men sitting around a table, all wearing traditional clothing with colorful fabrics and hats. they are all engaged in conversation and appear to be engaged in some kind of activity.",
num_inference_steps=30, embedded_guidance=3.5
)
image.save("image_with_lora.jpg")`

使用代码 可以生成 lora风格的图片 但是放到 comfyui中 同样的提示词 生成的图片 和lora 基本不相关,还需要其他什么操作么

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported difference using the supplied ModelManager.load_lora example and the equivalent ComfyUI workflow. Compare how the ckpt LoRA is loaded in both paths, then define the required loading or conversion steps and verify that the generated result reflects the LoRA style.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.