[QWEN3_OMNI] The Training of Qwen3-Omni-30B-A3B-Instruct is extremely slow

Describe the bug
the Training of Qwen/Qwen3-Omni-30B-A3B-Instruct is incredibly slow (250s/it).

Your hardware and system info

OS/kernel: Linux 6.8.0-1013-nvidia-64k aarch64 (Ubuntu 22.04-based).
Driver: NVIDIA 570.148.08 (aarch64) per /proc/driver/nvidia/version.
GPU (from lspci): GH200 (96GB)
CUDA toolkit/runtime (from installed packages): cu12.8 (e.g., nvidia-cuda-runtime-cu12==12.8.90, nvidia-cudnn-cu12==9.10.2.21).
PyTorch: 2.9.0+cu128
Python: (3.11).
ms-swift: 3.12.0.dev0
transformers: 4.57.3

the training config:

MAX_PIXELS=1003520 \
VIDEO_MAX_PIXELS=50176 \
ENABLE_AUDIO_OUTPUT=false \
FPS_MAX_FRAMES=12 \
CUDA_VISIBLE_DEVICES=0 \
OMP_NUM_THREADS=1 \ # it hangs using any other value
swift sft \
    --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
    --model_type qwen3_omni \
    --dataset '/home/ubuntu/ms-swift/test_10k.jsonl' \
    --split_dataset_ratio 0.01 \
    --load_from_cache_file true \
    --lazy_tokenize false \
    --train_type lora \
    --use_dora true \
    --lora_rank 16 \
    --lora_alpha 64 \
    --target_modules all-linear \
    --torch_dtype bfloat16 \
    --num_train_epochs 2 \
    --per_device_train_batch_size 2 \
    --per_device_eval_batch_size 1 \
    --attn_impl flash_attn \
    --learning_rate 1e-4 \
    --freeze_vit true \
    --freeze_aligner true \
    --packing true \
    --gradient_accumulation_steps 4 \
    --gradient_checkpointing true \
    --eval_steps 100 \
    --save_steps 500 \
    --save_total_limit 2 \
    --logging_steps 10 \
    --max_length 2048 \
    --warmup_ratio 0.05 \
    --dataset_num_proc 1 \
    --dataloader_num_workers 14 \
    --output_dir output/test

Additional context
Regardless of what i use, LoRA, DoRA, Full finetuning etc, any deepspeed config (or not using any deepspeed at all) the issue persists.
GH200 is as high end as it goes. the throughput shouldn't be this slow.
GPU Utilization hovers around 16-21%

I tried to test the megatron backend (I thought maybe that'd do something, as counter-intuitive as it sounds on a single gpu setup) but i always got :

[rank0]: RuntimeError: /TransformerEngine/transformer_engine/common/transformer_engine.cpp:314 in function Allocate: Cannot allocate a new NVTETensor. Maximum number of tensors reached: 70849. There is probably a memory leak in your application.

I appreciate your input. Hopefully we'll get a smaller Omni model like before by the good guys at Alibaba.

Start by reproducing the reported swift sft command with the Qwen/Qwen3-Omni-30B-A3B-Instruct configuration and compare GPU utilization and step time. Then investigate the Megatron backend path and the TransformerEngine tensor-allocation error; done means the performance bottleneck or allocation failure is identified and a verified fix or actionable diagnosis is documented.