invoke-ai / invoke-ai/InvokeAI
[enhancement]: Standardize Quantized Models on GGUF q4_k_m
- Dominant language
- Python
- Stars
- 28.2k
- Forks
- 3k
- Avg merge
- 6d 5h
- Merged PRs (30d)
- 19
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Contact Details
_No response_
### What should this feature add?
Now that Unsloth GGUFs have been rolled out for the quantized versions of [Flux Krea](https://github.com/invoke-ai/InvokeAI/blob/c5069557f385648005bcb76b3fdee9e29adc3f8f/invokeai/backend/model_manager/starter_models.py#L167) and [Flux Kontext](https://github.com/invoke-ai/InvokeAI/blob/c5069557f385648005bcb76b3fdee9e29adc3f8f/invokeai/backend/model_manager/starter_models.py#L151), you may wish to standardize the original Flux Quantized models on GGUF q4_k_m as well. This will be of particular benefit for users with GPUs that are incompatible with BitsandBytes, such as MPS (OSX) and AMD ROCm.
The relevant Unsloth model links are:
- [Schnell](https://huggingface.co/unsloth/FLUX.1-schnell-GGUF/resolve/main/flux1-schnell-Q4_K_M.gguf)
- [Dev](https://huggingface.co/unsloth/FLUX.1-dev-GGUF/resolve/main/flux1-dev-Q4_K_M.gguf)
### Alternatives
You may also wish to consider moving back to BitsandBytes once support for MPS and ROCm architectures, both of which are planned, matures. BnB should be faster than GGUF and may also use less VRAM. The NF4v2 Flux Dev checkpoint for Forge runs with as little as 6GB of VRAM.
### Additional Content
_No response_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.