huggingface / huggingface/diffusers

how to load lora weight with fp8 transfomer model?

Aperta
#11,648 23 commenti 0 reazioni 0 assegnatari Vedi su GitHub
stale
Lingua principale
Python
Stelle
34.5k
Fork
7.3k
Merge medio
3g 3h
PR unite (30g)
91

Descrizione

Hi, I want to run fluxcontrolpipeline with transformer_fp8 reference the code :
https://huggingface.co/docs/diffusers/api/pipelines/flux#quantization

```
import torch
from diffusers import BitsAndBytesConfig as DiffusersBitsAndBytesConfig, FluxTransformer2DModel, FluxControlPipeline
from transformers import BitsAndBytesConfig as BitsAndBytesConfig, T5EncoderModel

quant_config = BitsAndBytesConfig(load_in_8bit=True)
text_encoder_8bit = T5EncoderModel.from_pretrained(
"black-forest-labs/FLUX.1-dev",
subfolder="text_encoder_2",
quantization_config=quant_config,
torch_dtype=torch.float16,
)

quant_config = DiffusersBitsAndBytesConfig(load_in_8bit=True)
transformer_8bit = FluxTransformer2DModel.from_pretrained(
"black-forest-labs/FLUX.1-dev",
subfolder="transformer",
quantization_config=quant_config,
torch_dtype=torch.float16,
)

pipeline = FluxControlPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
text_encoder_2=text_encoder_8bit,
transformer=transformer_8bit,
torch_dtype=torch.float16,
device_map="balanced",
)

prompt = "a tiny astronaut hatching from an egg on the moon"
image = pipeline(prompt, guidance_scale=3.5, height=768, width=1360, num_inference_steps=50).images[0]
image.save("flux.png")
```

but when I load lora after build a pipeline

```
pipeline = FluxControlPipeline.from_pretrained(
"black-forest-labs/FLUX.1-dev",
text_encoder_2=text_encoder_8bit,
transformer=transformer_8bit,
torch_dtype=torch.float16,
device_map="balanced",
)

pipe.load_lora_weights("black-forest-labs/FLUX.1-Depth-dev-lora")
```
There a error:
not support fp8 weight , how to fix it??

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

The reported entry points are FluxControlPipeline.from_pretrained and load_lora_weights, with the quantized FluxTransformer2DModel setup shown in the issue. First reproduce the fp8-weight error using the linked Hugging Face quantization example and the FLUX.1-Depth-dev-lora reference; done means the supported loading path is identified or the limitation is documented.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
python, pytorch
Ambito
machine-learning
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
35/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.