invoke-ai / invoke-ai/InvokeAI

[enhancement]: support lodestones/Chroma

Open
#7,986 2 comments 15 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
28.2k
Forks
3k
Avg merge
6d 5h
Merged PRs (30d)
19

Description

### Is there an existing issue for this?

- [x] I have searched the existing issues

### What should this feature add?

https://huggingface.co/lodestones/Chroma is derived from FLUX.1 [Schnell] with some changes to the model architecture making it about 25% smaller. It retains the Apache license.

### Additional Content

Currently the model manager is willing to install a Chroma GGUF, but running it fails like
> Missing key(s) in state_dict: "time_in.in_layer.weight", […],
> Unexpected key(s) in state_dict: "distilled_guidance_layer.in_proj.bias", […]

### Checklist

Based on my experience with making a [third-party node for Chroma](https://gitlab.com/keturn/chroma_invoke), here are some things to keep in mind for a full integration:

- [ ] Uses **CFG** and not FLUX Guidance.
- [ ] Does not use CLIP. (You _can_ pass CLIP to it, because of its history from Schnell, but its not being trained with CLIP and is usable without it.)
- [ ] T5 encoder should not be either padded or cropped to 256/512 tokens…
- [ ] …but it's often useful to put padding tokens on the negative prompt, especially if it's not large on its own.
- [x] Should be able to use FLUX LoRA, with some tolerance as in #8059.
- [ ] Should also be able to use Chroma-specific LoRA.
- [ ] `torch.compile` is much more worthwhile than it was with SDXL. (Then this is likely true of FLUX too? Haven't checked.)
- [ ] Can use FLUX Redux. (But Flex.1 Redux looks pretty bad.)
- [ ] ControlNet - TBD

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.