Training hangs indefinitely after rollout phase completes on 8 B200 GPUs with CP=4, TP=2

Description

Training gets stuck indefinitely after the rollout phase completes and the training phase starts. The system shows only SGLang health check pings for hours with no training progress.

Environment

Hardware: 8× NVIDIA B200 GPUs
Container: Slime official container
Model: Qwen3-4B

Configuration

# Parallelism settings
--tensor-model-parallel-size 2
--sequence-parallel
--pipeline-model-parallel-size 1
--context-parallel-size 4
--expert-model-parallel-size 1
--expert-tensor-parallel-size 1

# Rollout settings
--rollout-batch-size 16
--n-samples-per-prompt 64
--rollout-max-response-len 40000
--num-steps-per-rollout 1

# Training settings
--colocate
--use-dynamic-batch-size
--max-tokens-per-gpu 32768
--calculate-per-token-loss
--use-slime-router

# SGLang settings
--rollout-num-gpus-per-engine 1
--sglang-mem-fraction-static 0.7

Expected Behavior

Training should proceed after rollout phase completes, showing training metrics and gradient updates.

Actual Behavior

The rollout phase completes successfully with metrics logged:

[2026-01-23 17:02:54] rollout 16: {'rollout/response_lengths': 18408.32, 'rollout/rewards': -8.03e-09, ...}
[2026-01-23 17:02:54] passrate 16: {'passrate/pass@1': 0.243, 'passrate/pass@2': 0.324, ...}
[2026-01-23 17:02:54] Timer actor_train start

After actor_train start, training hangs indefinitely. Only SGLang health check pings appear in the logs:

[2026-01-23 17:03:10] INFO: 172.17.0.2:43964 - "GET /health HTTP/1.1" 200 OK [repeated 8x across cluster]
[2026-01-23 17:03:40] INFO: 172.17.0.2:53212 - "GET /health HTTP/1.1" 200 OK [repeated 8x across cluster]
... (continues for hours with no training progress)
[2026-01-23 17:28:10] INFO: 172.17.0.2:42272 - "GET /health HTTP/1.1" 200 OK [repeated 8x across cluster]

The system remains in this state indefinitely with no training updates.

Additional Notes

• The issue is intermittent - encountered multiple times with different training configurations, but not 100% reproducible
• Rollout phase always completes successfully; the hang occurs specifically at the training phase start
• The hang appears to be related to the actor_train step, possibly a deadlock in distributed training communication or weight synchronization
• Health checks indicate SGLang engines remain alive/healthy during the hang
• No OOM issues are observed

Start by reproducing the transition from the completed rollout to the actor_train step with the listed CP=4 and TP=2 configuration. Inspect the distributed training communication and weight-synchronization path around actor_train; done means training proceeds past that transition and reports metrics and gradient updates instead of hanging while SGLang health checks continue.