[Bug] UpdateWeightFromDistributed fails for VLM models in non-colocate mode

Bug Description

When running VLM (Vision-Language Model) training without --colocate, the weight update from training to SGLang produces corrupted weights, causing SGLang to crash on the first generate request after weight sync.

The root cause is that UpdateWeightFromDistributed uses the manual convert_to_hf() function (regex-based name conversion) instead of the megatron-bridge for weight conversion. The bridge correctly handles all VLM parameters (vision encoder + language model), while the manual converter does not produce correct weights for the SGLang load_weights() path.

This is not an async-specific issue. Any non-colocate VLM configuration will trigger it, because:

• --colocate → UpdateWeightFromTensor → uses HfWeightIteratorBridge → correct
• no --colocate → UpdateWeightFromDistributed → uses convert_to_hf() → crash

Text-only models (e.g., Qwen3-4B) work fine in both modes because their convert_to_hf() implementation is correct.

Steps to Reproduce

Run the following script which is a non-colocate version of examples/geo3k_vlm/run_geo3k_vlm.sh:

bash examples/geo3k_vlm/run_geo3k_vlm_non_colocate.sh

The only difference from the working run_geo3k_vlm.sh is replacing:

# Colocate (working):
MISC_ARGS=(
   --colocate
)
--actor-num-gpus-per-node ${NUM_GPUS}

with:

# Non-colocate (crashes):
MISC_ARGS=(
   --actor-num-nodes 1
   --actor-num-gpus-per-node $((NUM_GPUS / 2))
   --rollout-num-gpus $((NUM_GPUS / 2))
)

The crash happens at the initial actor_model.update_weights() call, before the first rollout.

Expected Behavior

Non-colocate VLM training should work the same as colocate mode — weight updates should produce valid weights that SGLang can use for inference.

Actual Behavior

TODO

Environment

• slime version: latest (commit 1e99f92)
• Python version: 3.12
• PyTorch version: 2.x
• CUDA version: 12.x
• GPU type and count: 8x H100
• OS: Linux
• SGLang version: latest
• Megatron-LM version: latest

Logs

Training side completes weight update successfully (Qwen3-VL-8B-Instruct example):

[2026-03-07 00:07:00] timer.py:24 - Timer update_weights start
[2026-03-07 00:07:00] memory_utils.py:41 - [Rank 0] Memory-Usage before update_weights: {'gpu': '0', 'total_GB': 79.19, 'free_GB': 50.83, 'used_GB': 28.36}
[2026-03-07 00:07:03] memory_utils.py:41 - [Rank 0] Memory-Usage after update_weights: {'gpu': '0', 'total_GB': 79.19, 'free_GB': 46.43, 'used_GB': 32.76}
[2026-03-07 00:07:03] timer.py:32 - Timer update_weights end (elapsed: 3.5s)


SGLang side receives the update and acknowledges:

POST /update_weights_from_distributed HTTP/1.1 200 OK  (multiple calls, all 200)
POST /continue_generation HTTP/1.1 200 OK


Then all generate requests fail:

POST /generate HTTP/1.1 500 Internal Server Error  (repeated)

Additional Context

Verified working configurations

Configuration	Weight update class	Conversion method	Result
VLM + colocate	UpdateWeightFromTensor	HfWeightIteratorBridge	Works
Text + non-colocate	UpdateWeightFromDistributed	convert_qwen2_to_hf()	Works
VLM + non-colocate (8B-Instruct)	UpdateWeightFromDistributed	convert_qwen3vl_to_hf()	500 error
VLM + non-colocate (4B-Thinking)	UpdateWeightFromDistributed	convert_qwen3vl_to_hf()	NaN crash

Key code paths

The branching point is in slime/backends/megatron_utils/actor.py:

update_weight_cls = UpdateWeightFromTensor if self.args.colocate else UpdateWeightFromDistributed

• UpdateWeightFromTensor (colocate): uses HfWeightIteratorBridge which leverages megatron-bridge for correct architecture-aware weight conversion
• UpdateWeightFromDistributed (non-colocate): uses convert_to_hf() → convert_qwen3vl_to_hf(), a manual regex-based converter

Note that UpdateWeightFromTensor can also broadcast weights via NCCL to overflow engines (see _send_hf_params()), and this works because it sends bridge-converted HF weights. The NCCL broadcast mechanism itself is fine — the issue is purely in the weight conversion.

Potential fix direction

Make UpdateWeightFromDistributed use HfWeightIteratorBridge (or equivalent bridge-based conversion) instead of the manual convert_to_hf() for VLM models.

Pre-submission Checklist

• I have read the CONTRIBUTING.md and understand the collaboration scope.
• I have read the documentation and my issue is not addressed there.
• I have searched for existing issues and this is not a duplicate.
• I have provided a minimal, reproducible example.

Start at slime/backends/megatron_utils/actor.py and trace the UpdateWeightFromDistributed path into convert_to_hf() and convert_qwen3vl_to_hf(). Reproduce with examples/geo3k_vlm/run_geo3k_vlm_non_colocate.sh, comparing the working colocate configuration. Done means non-colocate VLM weight synchronization completes and subsequent SGLang generate requests do not return errors or NaNs.