Cuda Out of memory with DKMPalettizer
- Dominant language
- Python
- Stars
- 5.4k
- Forks
- 850
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 10
Description
## ❓Question
I'm trying to run training-time palletization with n_bits=4.
I always get an out-of-memory error on the first step (despite reducing batch size down to one.)
A few details:
- I'm quantising the stable diffusion v1.5 model.
- I have 24GB of GPU RAM.
Are there any tips to avoid this?
```
config = DKMPalettizerConfig(global_config=ModuleDKMPalettizerConfig(n_bits=4))
palettizer = DKMPalettizer(unet, config)
unet = palettizer.prepare(inplace=True)
unet, optimizer, lr_scheduler = accelerator.prepare(unet, optimizer, lr_scheduler)
```
Contributor guide
Research direction
Start by reproducing the training-time palletization flow with DKMPalettizer.prepare and accelerator.prepare using n_bits=4 on the stable diffusion v1.5 UNet. Measure the first-step GPU allocation and determine whether a documented workaround or a code change is needed; done means training proceeds without the out-of-memory failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100