deepseek-ai / deepseek-ai/Janus

Does Janus' image generation mode support multi-modal input?

Open
#221 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
17.8k
Forks
2.2k
PR merge metrics
No merged PRs in 30d

Description

Hi @learningpro @gnobitab @TheOneTrueGuy @mowentian @soloice , thanks for your excellent work!

I recently tried to use Janus for **image generation**.

In addition to providing text prompts, I also provided **other images as part of the prompt**. However, in the official implementation of Janus by transformers, I found the following code in the definition of **JanusProcessor**, which seems to mean that when generating images, the processor does not process the image elements in the input? So it seems that in image generation mode, the model only processes unimodal text prompts?

> # Process images if pixel values are provided.
> if images is not None and generation_mode != "image":
> data["pixel_values"] = self.image_processor(images=images, **output_kwargs["images_kwargs"])[
> "pixel_values"
> ]

Therefore, I hope to confirm with the official whether the image generation part of Janus supports **multimodal input**?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.