lllyasviel / lllyasviel/stable-diffusion-webui-forge

lllyasviel/flux1-dev-bnb-nf4 by Python

Open
#2,118 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
13k
Forks
1.7k
PR merge metrics
No merged PRs in 30d

Description

I try to go the
MODEL: lllyasviel/flux1-dev-bnb-nf4 from https://huggingface.co/lllyasviel/flux1-dev-bnb-nf4
and all the components:
VAE: https://huggingface.co/black-forest-labs/FLUX.1-dev/blob/main/ae.safetensors
ENCODER: https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/t5xxl_fp8_e4m3fn.safetensors
CLIP: https://huggingface.co/comfyanonymous/flux_text_encoders/blob/main/clip_l.safetensors

My repo now:

flux/
├── ae.safetensors
├── flux1-dev-bnb-nf4.safetensors
├── model_index.json   //made by me
├── tokenizer/
│   └── tokenizer.json   //from https://huggingface.co/black-forest-labs/FLUX.1-dev/tree/main/tokenizer_2
├── t5xxl_fp8/
 |   ├── config.json       //from https://huggingface.co/black-forest-labs/FLUX.1-dev/tree/main/transformer
│   └── model.safetensors
├── vae/                       
│   ├── config.json       //from here https://huggingface.co/black-forest-labs/FLUX.1-dev/tree/main/vae
│   └── diffusion_pytorch_model.safetensors
└── clip_l/
    ├── config.json
    └── model.safetensors

my code:

import os
import torch
from diffusers import AutoencoderKL, FluxTransformer2DModel, FluxPipeline
from transformers import CLIPTextModel, PreTrainedTokenizerFast, T5ForConditionalGeneration,logging

#print(os.path.exists(diffusion_model_path))  # Должно вывести True, если файл существует
#logging.set_verbosity(logging.DEBUG)
torch.cuda.empty_cache() 
torch.device('cuda')

#Components
print('vae')
vae = AutoencoderKL.from_pretrained("./flux/vae")

print('clip_l')
text_encoder = CLIPTextModel.from_pretrained("./flux/clip_l")

print('tokenizer')
tokenizer = PreTrainedTokenizerFast.from_pretrained("./flux/tokenizer")

print('t5xxl_fp8')
t5_model = T5ForConditionalGeneration.from_pretrained("./flux/t5xxl_fp8")

print('flux1-dev')
transformer = FluxTransformer2DModel.from_pretrained("./flux/flux1-dev-bnb-nf4.safetensors")

#Model
print('Create the pipeline with the loaded models...')
model = FluxPipeline(vae=vae, text_encoder=text_encoder, tokenizer=tokenizer, transformer=transformer, t5_model=t5_model)

The error:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[11], line 22
     19 tokenizer = PreTrainedTokenizerFast.from_pretrained("./flux/tokenizer")
     21 print('t5xxl_fp8')
---> 22 t5_model = T5ForConditionalGeneration.from_pretrained("./flux/t5xxl_fp8")
     24 print('flux1-dev')
     25 transformer = FluxTransformer2DModel.from_pretrained("./flux/flux1-dev-bnb-nf4.safetensors")

File ~/.local/lib/python3.12/site-packages/transformers/modeling_utils.py:3792, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, **kwargs)
   3789 with safe_open(resolved_archive_file, framework="pt") as f:
   3790     metadata = f.metadata()
-> 3792 if metadata.get("format") == "pt":
   3793     pass
   3794 elif metadata.get("format") == "tf":

AttributeError: 'NoneType' object has no attribute 'get'

Maybe somebody knows my mistake?
P.S. Im really bad with config.files

Thanks!!!!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the local paths in model_index.json and the ./flux/t5xxl_fp8 loading call, then compare the referenced config.json and model.safetensors with the Hugging Face model layout described in the issue. Check whether the checkpoint metadata is readable by T5ForConditionalGeneration; done means the component loads without the reported AttributeError and can be passed to the FluxPipeline.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.