Lightning-AI / Lightning-AI/pytorch-lightning
Fail to quantize SDXL model with lightning codes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
### Bug description
It seems not good to use BitsandbytesPrecision directly as shown at front pages of Lightning.
what should I do to quantize SDXL and make it saved after quantization?
Codes and bugs are shown below
### What version are you seeing the problem on?
v2.1
### How to reproduce the bug
```python
from lightning.fabric import Fabric
from lightning.fabric.plugins import BitsandbytesPrecision
from diffusers import StableDiffusionXLPipeline, StableDiffusionXLImg2ImgPipeline
import torch
import time
path = '/mnt/hub/models--stabilityai--stable-diffusion-xl-base-1.0/snapshots/bf714989e22c57ddc1c453bf74dab4521acb81d8/'
prompt = "hyperrealistic glamour portrait of an old weary wizard surrounded by elemental magic, arcane, freckles, skin pores, pores, velus hair, macro, extreme details, looking at viewer"
negative_prompt = "sketch, cartoon, drawing, anime:1.4, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions"
torch.set_grad_enabled(False)
torch.backends.cudnn.benchmark = True
with torch.inference_mode():
pipe = StableDiffusionXLPipeline.from_pretrained(
path, torch_dtype=torch.float16, use_safetensors=True
)
pipe.to(f"cuda")
pipe.unet.to(device=f"cuda", dtype=torch.float16, memory_format=torch.channels_last)
img = pipe(prompt=prompt,negative_prompt=negative_prompt, num_inference_steps=50, guidance_scale = 9, num_images_per_prompt=1).images[0]
img.save(f"image.png")
mode = "nf4"
plugin = BitsandbytesPrecision(mode=mode)
fabric = Fabric(plugins=plugin)
model = fabric.setup_module(pipe) # quantizes the layers
```
### Error messages and logs
Bugs are like this:
```
Traceback (most recent call last):
File "/tmp/lighting/quant.py", line 52, in
File "/opt/conda/lib/python3.9/site-packages/lightning/fabric/fabric.py", line 289, in setup_module
module = self._precision.convert_module(module)
File "/opt/conda/lib/python3.9/site-packages/lightning/fabric/plugins/precision/bitsandbytes.py", line 101, in convert_module
if not any(isinstance(m, torch.nn.Linear) for m in module.modules()):
File "/opt/conda/lib/python3.9/site-packages/diffusers/configuration_utils.py", line 137, in __getattr__
raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'StableDiffusionXLPipeline' object has no attribute 'modules'
```
### Environment
Current environment
```
#- Lightning Component (e.g. Trainer, LightningModule, LightningApp, LightningWork, LightningFlow):
#- PyTorch Lightning Version (e.g., 1.5.0):
#- Lightning App Version (e.g., 0.5.2):
#- PyTorch Version (e.g., 2.0):
#- Python version (e.g., 3.9):
#- OS (e.g., Linux):
#- CUDA/cuDNN version:
#- GPU models and configuration:
#- How you installed Lightning(`conda`, `pip`, source):
#- Running environment of LightningApp (e.g. local, cloud):
```
### More info
_No response_
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the script against the BitsandbytesPrecision path shown in lightning/fabric/plugins/precision/bitsandbytes.py, starting at Fabric.setup_module and its module conversion check. Determine how the SDXL pipeline should be passed for quantization and what behavior is expected for saving the quantized result; completion should include a documented or tested path that avoids the reported AttributeError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100