qwen3_moe_30b_hellaswag OOMs during model loading on single GPU

Describe the bug

qwen3_moe_30b_hellaswag fails to load on a single H100 GPU. For context, if we remove the nemo_automodel BackendConfig for the recipe and just use default HF, the model does load.

0:   File "/opt/Automodel/nemo_automodel/components/checkpoint/checkpointing.py", line 600, in load_base_model
0:     self.load_model(
0:   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 124, in decorate_context
0:     return func(*args, **kwargs)
0:            ^^^^^^^^^^^^^^^^^^^^^
0:   File "/opt/Automodel/nemo_automodel/components/checkpoint/checkpointing.py", line 457, in load_model
0:     state_dict = _maybe_adapt_state_dict_to_hf(
0:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0:   File "/opt/Automodel/nemo_automodel/components/checkpoint/checkpointing.py", line 1448, in _maybe_adapt_state_dict_to_hf
0:     return adapter.to_hf(state_dict, exclude_key_regex=r".*_extra_state.*", quantization=quantization, **kwargs)
0:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0:   File "/opt/Automodel/nemo_automodel/components/models/qwen3_moe/state_dict_adapter.py", line 64, in to_hf
0:     converted_tensors = self.convert_single_tensor_to_hf(
0:                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0:   File "/opt/Automodel/nemo_automodel/components/models/qwen3_moe/state_dict_adapter.py", line 103, in convert_single_tensor_to_hf
0:     expert_result = self._convert_single_merged_expert_to_hf_split_experts(fqn, tensor, **kwargs)
0:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0:   File "/opt/Automodel/nemo_automodel/components/moe/state_dict_mixin.py", line 555, in _convert_single_merged_expert_to_hf_split_experts
0:     w_up = w[:, inter_dim:].transpose(0, 1).contiguous()
0:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 79.11 GiB of which 4.50 MiB is free. Including non-PyTorch memory, this process has 79.09 GiB memory in use. Of the allocated memory 78.49 GiB is allocated by PyTorch, and 3.18 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Steps/Code to reproduce bug

Run the recipe on a single H100, but overwrite distributed.ep_size to 1.

Expected behavior

I expect the model to load. It may fail later in the training process if the batch size or sequence length is too large, but it's not even getting to that point.

Additional context

Internal NVIDIA log link: https://gitlab-master.nvidia.com/dl/jet/ci/-/jobs/302709974

Should you update the recipe, plz replace "you can potentially run it on less GPUs" with "you can potentially run it on fewer GPUs."