invoke-ai / invoke-ai/InvokeAI

[enhancement]: Standardize Quantized Models on GGUF q4_k_m

Open
#8,421 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
28.2k
Forks
3k
Avg merge
6d 5h
Merged PRs (30d)
19

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### Contact Details

_No response_

### What should this feature add?

Now that Unsloth GGUFs have been rolled out for the quantized versions of [Flux Krea](https://github.com/invoke-ai/InvokeAI/blob/c5069557f385648005bcb76b3fdee9e29adc3f8f/invokeai/backend/model_manager/starter_models.py#L167) and [Flux Kontext](https://github.com/invoke-ai/InvokeAI/blob/c5069557f385648005bcb76b3fdee9e29adc3f8f/invokeai/backend/model_manager/starter_models.py#L151), you may wish to standardize the original Flux Quantized models on GGUF q4_k_m as well. This will be of particular benefit for users with GPUs that are incompatible with BitsandBytes, such as MPS (OSX) and AMD ROCm.

The relevant Unsloth model links are:

- [Schnell](https://huggingface.co/unsloth/FLUX.1-schnell-GGUF/resolve/main/flux1-schnell-Q4_K_M.gguf)
- [Dev](https://huggingface.co/unsloth/FLUX.1-dev-GGUF/resolve/main/flux1-dev-Q4_K_M.gguf)

### Alternatives

You may also wish to consider moving back to BitsandBytes once support for MPS and ROCm architectures, both of which are planned, matures. BnB should be faster than GGUF and may also use less VRAM. The NF4v2 Flux Dev checkpoint for Forge runs with as little as 6GB of VRAM.

### Additional Content

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.