NVIDIA / NVIDIA/Megatron-LM

Integrate NeMo Automodel Qwen3.5-35B-A3B recipe with MFSDP v2

Open
#6,139 0 comments 0 reactions 1 assignee Claimed by @wujingyue View on GitHub
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

## Summary

The NeMo Automodel Qwen3.5-35B-A3B VLM fine-tuning recipe currently uses PyTorch FSDP2 and needs a validated Megatron Core path using experimental MFSDP v2.

Reference recipe:
- https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/qwen3_5_moe/qwen3_5_35b.yaml

## Reproduction / starting point

The reference recipe runs with:

```bash
automodel examples/vlm_finetune/qwen3_5_moe/qwen3_5_35b.yaml --nproc-per-node 8
```

Use the same model and the closest equivalent data, parallelism, and checkpoint settings through the MFSDP v2 path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.