NVIDIA-NeMo / NVIDIA-NeMo/Automodel

Add a VLM parity test for Qwen3.5-MoE

Open
#3,707 0 comments 0 reactions 1 assignee View on GitHub

@athitten is already working on this.

Since Aug 27, 2026.

Dominant language
Python
Stars
960
Forks
316
Avg merge
3d 20h
Merged PRs (30d)
143

Description

Follow-up to https://github.com/NVIDIA-NeMo/Automodel/pull/3649.

Context

PR #3649 adds PP and EP parity tests for Qwen3.5-MoE, but they run the text-only Qwen3_5MoeForCausalLM. All seven shipped recipes under examples/vlm_finetune/qwen3_5_moe/ use the VLM class Qwen3_5MoeForConditionalGeneration instead.

The text backbone was used because the VLM class needs an image processor to build its inputs, and there is no Qwen3.5 processor staged in CI's TEST_DATA_DIR. The text-only path lets the test generate its own token IDs and stage nothing.

What this leaves uncovered

Expert routing, expert sharding and the pipeline split all live in the text backbone, so those are covered today. Not covered:

  • the vision tower under PP and EP
  • VLM-specific pipeline routing (how pixel_values reach stage 0)
  • the VLM class's own config and stage-metadata contract

What to do

Stage a small Qwen3.5 processor in TEST_DATA_DIR, the way hf_gemma4_e4b_2l is staged for the gemma4 parity tests. That would let the proxy switch to NeMoAutoModelForImageTextToText.from_config and use the mock VLM dataset (components/datasets/vlm/mock.py), which generates random images and needs no real dataset.

Then either:

  • replace the text-only proxy with a VLM one, so the test matches the shipped recipes, or
  • add a VLM test alongside it, keeping the text one as cheaper coverage of the backbone.

Replacing is probably right unless we want both, since the shipped recipes are all VLM.

Note

Two smaller gaps in Qwen3_5MoeForCausalLM were worked around in the proxy and are worth checking if we keep it:

  • it does not declare _keep_in_fp32_modules_strict, so linear-attention layers fail under FSDP2 with FSDP expects uniform original parameter dtype (the VLM class declares it at model.py:1098)
  • setting pad_token_id gives the embedding a padding_idx, which under FSDP2 makes checkpointing.py skip weight initialization entirely, so a from_config model trains on uninitialized weights

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.