modelscope / modelscope/DiffSynth-Studio

DiffSynth-Studio低显存示例代码中的vram_config参数与ModelConfig不兼容导致报错

Open
#1,142 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13.1k
Forks
1.3k
Avg merge
13h 12m
Merged PRs (30d)
45

Description

非常感谢各位大佬做出的贡献,我想尝试在4070Super的显卡推理Qwen-Image-Edit-2509模型,为了节省显存,不至于显存爆炸而使用https://github.com/modelscope/DiffSynth-Studio/blob/main/examples/qwen_image/model_inference_low_vram/Qwen-Image-Edit-2509.py提供的样例代码,直接复制粘贴到pycharm后进行推理但是报错了。我查看了ModelConfig的源码,发现vram_config配置的参数,有很多参数均是无效的,vram_config参数如下:

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": torch.float8_e4m3fn,
    "onload_device": "cpu",
    "preparing_dtype": torch.float8_e4m3fn,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}

ModelConfig如下:

  path: Union[str, list[str]] = None
    model_id: str = None
    origin_file_pattern: Union[str, list[str]] = None
    download_resource: str = "ModelScope"
    offload_device: Optional[Union[str, torch.device]] = None
    offload_dtype: Optional[torch.dtype] = None
    local_model_path: str = None
    skip_download: bool = False

具体报错如下:

File "E:\Project_Style\Qwen-Image-Edit\Project\demo.py", line 25, in <module>
    ModelConfig(model_id="Qwen/Qwen-Image-Edit-2509", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors", **vram_config),
TypeError: ModelConfig.__init__() got an unexpected keyword argument 'onload_dtype'

我的环境中diffsynth 的Version: 1.1.9,我不知道哪个版本才可以运行官方提供的代码,如果有知道的朋友,劳烦您告诉我一声,非常的感谢!
后来为了解决该问题,并且不明白为什么使用Qwen-Image-Edit-2509进行推理,却还要加载Qwen-Image的权重文件,我在modelscope中提前下载好Qwen-Image-Edit-2509模型。尝试修改代码,但依旧无效并报错。
具体代码如下:

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",}

model_path=r"E:\Project_Style\Qwen-Image-Edit\Models"
model_id="Qwen/Qwen-Image-Edit-2509"
pipe = QwenImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id=model_id,
                    origin_file_pattern="transformer/",
                    local_model_path=model_path,
                    skip_download=True,
                    **vram_config
                    ),
        ModelConfig(model_id=model_id,
                    origin_file_pattern="text_encoder/",
                    local_model_path=model_path,
                    skip_download=True,
                    **vram_config
                    ),
        ModelConfig(model_id=model_id,
                    origin_file_pattern="vae/",
                    local_model_path=model_path,
                    skip_download=True,
                    **vram_config
                    ),],
    processor_config=ModelConfig(model_id=model_id,
                                 origin_file_pattern="processor/",
                                 skip_download=True,
                                 local_model_path=model_path,),

)

报错如下:

Traceback (most recent call last):
  File "E:\Project_Style\Qwen-Image-Edit\Project\降低显存的方式.py", line 28, in <module>
    pipe = QwenImagePipeline.from_pretrained(
  File "D:\Miniconda3\envs\llm\lib\site-packages\diffsynth\pipelines\qwen_image.py", line 331, in from_pretrained
    model_manager.load_model(
  File "D:\Miniconda3\envs\llm\lib\site-packages\diffsynth\models\model_manager.py", line 408, in load_model
    if model_detector.match(file_path, state_dict):
  File "D:\Miniconda3\envs\llm\lib\site-packages\diffsynth\models\model_manager.py", line 165, in match
    if len(state_dict) == 0:
TypeError: object of type 'NoneType' has no len()
Loading models from: E:\Project_Style\Qwen-Image-Edit\Models\Qwen/Qwen-Image-Edit-2509\transformer/

为了解决这些问题,我耗费了很长的时间依旧没有头绪,有朋友知道如何解决该问题的,我非常的愿意向您请教,在此提前向您表示感谢!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with examples/qwen_image/model_inference_low_vram/Qwen-Image-Edit-2509.py and compare its ModelConfig fields with the installed DiffSynth version 1.1.9. Then trace the loading path through qwen_image.py and model_manager.py, using the reported errors and local model path as reproduction points. Done means the example or documented local-model configuration runs without the reported TypeErrors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.