框架设置中权重offload和reload交换过程中显存超出ray的最大显存设置，导致ray主动杀进程的情况。

采用Qwen3-8B在4卡A800跑，之前也在H20上跑过，发现问题。当infer和train、reference放在相同的卡上面时。例如：
都是list(range(0,4)。模型offload和reload的交换过程中会爆掉显存，引起ray的主动杀进程。但是讲train、reference等放在0～2卡，infer放在3卡，相同的配置完全可以正常运行。能不能优化模型切换中的流程，降低切换过程中显存挤占现象。

我在长上下文情况下，80G的卡模型8B的也得tensor并行度为2、vllm也tensor并行2，才可以避免因降低切换过程中显存挤占现象导致的ray杀掉workers。但是实际切换完成之后内存占用的又不高，单独看infer运行只有40G，train、reference是50～60G。

4卡8B 2 tensor并行的配置如下(共享环境)：
hydra:
run:
dir: .
output_subdir: null

exp_name: "qwen3-8B-medical-rlvr"
seed: 42
logging_dir: ./outputs/logs
output_dir: ./outputs
system_envs:
USE_MODELSCOPE: '1'

checkpoint_config:
type: file_system
output_dir: /data/testmllm/project/ROLL/models/${exp_name}

track_with: tensorboard
tracker_kwargs:
log_dir: ./rl_examples/llm/tensorboard/roll_exp/${exp_name}

num_gpus_per_node: 4

max_steps: 21
save_steps: 10
logging_steps: 1
eval_steps: 100
resume_from_checkpoint: false

rollout_batch_size: 64 # prompt
prompt_length: 5500
response_length: 4096

num_return_sequences_in_group: 8
ppo_epochs: 1
adv_estimator: "grpo"
importance_sampling: seq
# use_kl_loss: true
# kl_loss_coef: 0.001
# loss_agg_mode: "seq-mean-token-mean"

# clip
value_clip: 0.5
reward_clip: 10
advantage_clip: 2.0
dual_clip_loss: true

# normalize
reward_norm: null
reward_shift: false
reward_scale: false

# data mask
max_len_mask: true
difficulty_mask: true
difficulty_low_threshold: 1.05
difficulty_high_threshold: 1.95
error_max_len_clip: false

# system_envs:
# RAY_PROFILING: "1"
# profiler_output_dir: ./llm/profile/${exp_name}

# data weight
difficulty_loss_weight: false
length_loss_weight: false

# reward
add_token_level_kl: false

# advantage
whiten_advantages: true

# dynamic sampling scheduler
# use_additional_prompts: true
# max_running_requests: 256
# is_num_return_sequences_expand: false

pretrain: /data/testmllm/models/Qwen/Qwen3-8B
reward_pretrain: /data/testmllm/models/Qwen/Qwen3-8B

validation:
data_args:
template: qwen3
file_name:
- medical_rl/dataset/eval.jsonl
generating_args:
max_new_tokens: ${response_length}
top_p: 0.6
top_k: 50
num_beams: 1
temperature: 0.6
num_return_sequences: 1

actor_train:
model_args:
# attn_implementation: fa2
disable_gradient_checkpointing: false
dtype: bf16
model_type: ~
training_args:
learning_rate: 5.0e-6
weight_decay: 0
per_device_train_batch_size: 1
gradient_accumulation_steps: 16
warmup_steps: 20
num_train_epochs: 50
data_args:
template: qwen3
file_name:
- medical_rl/dataset/train.jsonl
domain_interleave_probs:
medical: 1
dataset_dir: data
messages: messages
interleave_probs: "1.0"
preprocessing_num_workers: 8
strategy_args:
strategy_name: megatron_train
strategy_config:
tensor_model_parallel_size: 2
pipeline_model_parallel_size: 1
expert_model_parallel_size: 1
use_distributed_optimizer: true
recompute_granularity: full
device_mapping: list(range(0,4))
infer_batch_size: 2

actor_infer:
model_args:
disable_gradient_checkpointing: true
dtype: bf16
generating_args:
max_new_tokens: ${response_length}
top_p: 0.99
top_k: 100
num_beams: 1
temperature: 0.99
num_return_sequences: ${num_return_sequences_in_group}
data_args:
template: qwen3
strategy_args:
strategy_name: vllm
strategy_config:
gpu_memory_utilization: 0.6
block_size: 16
max_model_len: 10000
tensor_parallel_size: 2
device_mapping: list(range(0,4))
infer_batch_size: 4

reference:
model_args:
disable_gradient_checkpointing: true
dtype: bf16
model_type: ~
data_args:
template: qwen3
strategy_args:
strategy_name: megatron_infer
strategy_config:
tensor_model_parallel_size: 1
pipeline_model_parallel_size: 1
expert_model_parallel_size: 1
device_mapping: list(range(0,4))
infer_batch_size: 2

rewards:
medical:
worker_cls: medical_rl.rewards.medical_rule_reward_worker.MedicalRuleRewardWorker
model_args:
model_name_or_path: ${reward_pretrain}
data_args:
template: qwen3
tag_included: [medical]
world_size: 1
infer_batch_size: 2

4卡8B 无模型并行的配置如下(infer和train隔离环境)：
hydra:
run:
dir: .
output_subdir: null

exp_name: "qwen3-8B-medical-rlvr"
seed: 42
logging_dir: ./outputs/logs
output_dir: ./outputs
system_envs:
USE_MODELSCOPE: '1'

checkpoint_config:
type: file_system
output_dir: /data/testmllm/project/ROLL/models/${exp_name}

track_with: tensorboard
tracker_kwargs:
log_dir: ./rl_examples/llm/tensorboard/roll_exp/${exp_name}

num_gpus_per_node: 4

max_steps: 21
save_steps: 10
logging_steps: 1
eval_steps: 100
resume_from_checkpoint: false

rollout_batch_size: 64 # prompt
prompt_length: 5500
response_length: 4096

num_return_sequences_in_group: 8
ppo_epochs: 1
adv_estimator: "grpo"
importance_sampling: seq
# use_kl_loss: true
# kl_loss_coef: 0.001
# loss_agg_mode: "seq-mean-token-mean"

# clip
value_clip: 0.5
reward_clip: 10
advantage_clip: 2.0
dual_clip_loss: true

# normalize
reward_norm: null
reward_shift: false
reward_scale: false

# data mask
max_len_mask: true
difficulty_mask: true
difficulty_low_threshold: 1.05
difficulty_high_threshold: 1.95
error_max_len_clip: false

# system_envs:
# RAY_PROFILING: "1"
# profiler_output_dir: ./llm/profile/${exp_name}

# data weight
difficulty_loss_weight: false
length_loss_weight: false

# reward
add_token_level_kl: false

# advantage
whiten_advantages: true

# dynamic sampling scheduler
# use_additional_prompts: true
# max_running_requests: 256
# is_num_return_sequences_expand: false

pretrain: /data/testmllm/models/Qwen/Qwen3-8B
reward_pretrain: /data/testmllm/models/Qwen/Qwen3-8B

validation:
data_args:
template: qwen3
file_name:
- medical_rl/dataset/eval.jsonl
generating_args:
max_new_tokens: ${response_length}
top_p: 0.6
top_k: 50
num_beams: 1
temperature: 0.6
num_return_sequences: 1

actor_train:
model_args:
# attn_implementation: fa2
disable_gradient_checkpointing: false
dtype: bf16
model_type: ~
training_args:
learning_rate: 5.0e-6
weight_decay: 0
per_device_train_batch_size: 1
gradient_accumulation_steps: 16
warmup_steps: 20
num_train_epochs: 50
data_args:
template: qwen3
file_name:
- medical_rl/dataset/train.jsonl
domain_interleave_probs:
medical: 1
dataset_dir: data
messages: messages
interleave_probs: "1.0"
preprocessing_num_workers: 8
strategy_args:
strategy_name: megatron_train
strategy_config:
tensor_model_parallel_size: 1
pipeline_model_parallel_size: 1
expert_model_parallel_size: 1
use_distributed_optimizer: true
recompute_granularity: full
device_mapping: list(range(0,3))
infer_batch_size: 2

actor_infer:
model_args:
disable_gradient_checkpointing: true
dtype: bf16
generating_args:
max_new_tokens: ${response_length}
top_p: 0.99
top_k: 100
num_beams: 1
temperature: 0.99
num_return_sequences: ${num_return_sequences_in_group}
data_args:
template: qwen3
strategy_args:
strategy_name: vllm
strategy_config:
gpu_memory_utilization: 0.6
block_size: 16
max_model_len: 10000
tensor_parallel_size: 1
device_mapping: list(range(3,4))
infer_batch_size: 4

reference:
model_args:
disable_gradient_checkpointing: true
dtype: bf16
model_type: ~
data_args:
template: qwen3
strategy_args:
strategy_name: megatron_infer
strategy_config:
tensor_model_parallel_size: 1
pipeline_model_parallel_size: 1
expert_model_parallel_size: 1
device_mapping: list(range(0,3))
infer_batch_size: 2

rewards:
medical:
worker_cls: medical_rl.rewards.medical_rule_reward_worker.MedicalRuleRewardWorker
model_args:
model_name_or_path: ${reward_pretrain}
data_args:
template: qwen3
tag_included: [medical]
world_size: 1
infer_batch_size: 2

Start by reproducing the reported peak-memory failure with the actor_train, actor_infer, and reference configurations, comparing shared device_mapping settings with the separated setup. Trace the offload/reload transition and Ray worker termination; done means the shared-card configuration completes model switching without exceeding the configured GPU memory limit.

Written by the indexing model from the issue text.