Comfy-Org / Comfy-Org/ComfyUI

would it be possible to try to add DeepSeek Janus Pro's text_encode configuration

Open
#6,846 0 comments 0 reactions 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 6h
Merged PRs (30d)
155

Description

### Feature Idea

Since DeepSeek Janus Pro has very good image comprehension, would it be possible to try to add Janus Pro's text_encode configuration and use it to comprehend the image and adjust the ClipTextEncode appropriately before handing it over to the Flux model for image modification, perhaps getting a strong Edit effect boost?

### Existing Solutions

Currently analyzed Janus Pro's model, using llama's tokenizer, but the number of layers and the shape of each layer is a bit confusing as to how ComfyUI converts it.

I made my own object parser to parse Janus Pro's model objects (the two core models, their respective layers and the torch module they call) see below:
https://github.com/comfyanonymous/ComfyUI/issues/6640#issuecomment-2635637873

### Other

I put it directly into the CLIP Extra Models directory, and was able to execute it smoothly, but I can find that the text does not present a successful CLIP. so I think that I should need to match up the different layers, but I haven't messed with this problem at all, and it's really a bit of a headache.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.