Support unified understanding + generation models (Janus-Pro, Show-o)

Hi NeMo AutoModel team,

First of all, thank you very much for the excellent work on the project — the unified VLM / MoE / Omni infrastructure has made it remarkably easy to fine-tune frontier multimodal models.

I noticed that the framework currently supports two largely separate worlds:

• Understanding-only VLMs/Omnis (Qwen3-VL, KimiVL, Mistral3, Gemma 3/4, Qwen3-Omni, Nemotron-Omni, …) under NeMoAutoModelForImageTextToText, and
• Pure generation diffusion models (FLUX, Wan 2.1, HunyuanVideo, Qwen-Image) under NeMoAutoDiffusionPipeline.

A new model family is now starting to bridge those two worlds — a single autoregressive transformer that can both understand images and generate them, without a separate diffusion pipeline. Two openly released checkpoints today actually do both in one model:

(For reference, similar architectures exist but are not currently good fine-tuning targets in this category: Meta's Chameleon released its checkpoint with the image-generation head removed, and BAAI's Emu3 ships as two separate Emu3-Gen and Emu3-Chat fine-tunes rather than a single unified checkpoint. They might inform design but are not on the integration list.)

I would like to open this issue to:

1. Ask whether unified understanding-and-generation models are on the roadmap. They sit somewhere between today's ForConditionalGeneration VLMs (which only emit text) and today's diffusion pipelines (which only generate). Adding them would also surface useful general-purpose primitives — image tokenizer / detokenizer integration, a unified vocabulary head, and a training loop that mixes language CE loss with image-token loss.

2. Highlight why this would be a meaningful direction for the framework. Janus-Pro is widely cited as a likely path toward GPT-4o-style native multimodal generation; supporting it would put NeMo AutoModel ahead of most open-source training frameworks (today essentially only LLaMA-Factory and a few research forks have partial support, and none have production-grade FSDP2 / TP / PP). Once the abstractions exist, follow-on models like Janus-Flow, OmniGen, or the next wave of unified architectures would mostly be config + adapter work rather than new infrastructure.

3. Offer to contribute. I would be happy to take this on, starting with Janus-Pro as the smallest concrete target (well-documented model, clean state dict, and the dual-encoder design exercises every interesting integration point). A reasonable plan looks like:

   • Add a nemo_automodel.components.models.janus/ (model + state-dict adapter + tokenizer integration for the VQ image head).
   • Add a recipes/multimodal_generate/ (or extend recipes/vlm/) with a training loop that supports interleaved image/text tokens and joint loss.
   • Wire a registry entry under MODEL_ARCH_MAPPING and a NeMoAutoModelForUnifiedMultimodal (name TBD) auto-class.
   • Add unit tests (forward shape, state-dict round-trip, layer equivalence vs HF reference) and at least one functional fine-tuning recipe + golden-loss check.

Before starting, I would love to hear:

• Is this direction interesting for the project?
• Are there architectural conventions I should align with — e.g., should unified models live under the existing NeMoAutoModelForImageTextToText umbrella with an extra capability flag, or get their own auto-class?
• Are there preferences on where the new auto-class and recipe should live in the directory tree?

Thanks again for the great work, and looking forward to your guidance on whether this fits the roadmap.