Loading Qwen/Qwen3.8-27B failed with NeMoAutoModelForCausalLM(OOM), but successfully load with Qwen3_5ForCausalLM

Describe the bug

When loading Qwen/Qwen3.8-27B with NeMoAutoModelForCausalLM.from_pretrained() on L20(40G), raise OOM

OutOfMemoryError: CUDA out of memory. Tried to allocate 170.00 MiB. GPU 0 has a total capacity of 44.53 GiB of which 33.94 MiB is free. Process 1844672 has 44.49 GiB memory in use. Of the allocated memory 44.19 GiB is allocated by PyTorch, and 21.40 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation.  See documentation for Memory Management  (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

When loading Qwen/Qwen3.8-27B with Qwen3_5ForCausalLM.from_pretrained() on same device, it is very slow but successed.

Steps/Code to reproduce bug

from nemo_automodel import NeMoAutoModelForCausalLM
model = NeMoAutoModelForCausalLM.from_pretrained("Qwen/Qwen3.8-27B/")

from nemo_automodel.components.models.qwen3_5.model import Qwen3_5ForCausalLM
model = Qwen3_5ForCausalLM.from_pretrained("Qwen/Qwen3.8-27B/")
rope_fusion is temporarily force-disabled globally (see #3027).
[transformers] `torch_dtype` is deprecated! Use `dtype` instead!
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d
[transformers] The fast path is not available because one of the required library is not installed. Falling back to torch implementation. To install follow https://github.com/fla-org/flash-linear-attention#installation and https://github.com/Dao-AILab/causal-conv1d

torch==2.8.0
nemo_automodel==0.6.0
transformer_engine==2.14.0

Expected behavior

A clear and concise description of what you expected to happen.

Additional context

Add any other context about the problem here.