NVIDIA-NeMo / NVIDIA-NeMo/Automodel
Add a VLM parity test for Qwen3.5-MoE
@athitten is already working on this.
Since Aug 27, 2026.
- Dominant language
- Python
- Stars
- 960
- Forks
- 316
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Follow-up to https://github.com/NVIDIA-NeMo/Automodel/pull/3649.
Context
PR #3649 adds PP and EP parity tests for Qwen3.5-MoE, but they run the text-only Qwen3_5MoeForCausalLM. All seven shipped recipes under examples/vlm_finetune/qwen3_5_moe/ use the VLM class Qwen3_5MoeForConditionalGeneration instead.
The text backbone was used because the VLM class needs an image processor to build its inputs, and there is no Qwen3.5 processor staged in CI's TEST_DATA_DIR. The text-only path lets the test generate its own token IDs and stage nothing.
What this leaves uncovered
Expert routing, expert sharding and the pipeline split all live in the text backbone, so those are covered today. Not covered:
- the vision tower under PP and EP
- VLM-specific pipeline routing (how
pixel_valuesreach stage 0) - the VLM class's own config and stage-metadata contract
What to do
Stage a small Qwen3.5 processor in TEST_DATA_DIR, the way hf_gemma4_e4b_2l is staged for the gemma4 parity tests. That would let the proxy switch to NeMoAutoModelForImageTextToText.from_config and use the mock VLM dataset (components/datasets/vlm/mock.py), which generates random images and needs no real dataset.
Then either:
- replace the text-only proxy with a VLM one, so the test matches the shipped recipes, or
- add a VLM test alongside it, keeping the text one as cheaper coverage of the backbone.
Replacing is probably right unless we want both, since the shipped recipes are all VLM.
Note
Two smaller gaps in Qwen3_5MoeForCausalLM were worked around in the proxy and are worth checking if we keep it:
- it does not declare
_keep_in_fp32_modules_strict, so linear-attention layers fail under FSDP2 withFSDP expects uniform original parameter dtype(the VLM class declares it atmodel.py:1098) - setting
pad_token_idgives the embedding apadding_idx, which under FSDP2 makescheckpointing.pyskip weight initialization entirely, so afrom_configmodel trains on uninitialized weights
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.