GKD cuda访问越界

Checklist / 检查清单

• I have searched existing issues, and this is a new bug report. / 我已经搜索过现有的 issues，确认这是一个新的 bug report。

Bug Description / Bug 描述

进行GKD（on-policy distillation）训练（student rollout+student training+teacher vllm server）时，发现偶尔会报错cuda illegal（无法稳定复现），想请问这个可能是什么原因？
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: Traceback (most recent call last):
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/yangpenghui1/ms-swift/swift/cli/rlhf.py", line 7, in
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: rlhf_main()
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/yangpenghui1/ms-swift/swift/pipelines/train/rlhf.py", line 246, in rlhf_main
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return SwiftRLHF(args).main()
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/yangpenghui1/ms-swift/swift/pipelines/base.py", line 52, in main
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: result = self.run()
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/yangpenghui1/ms-swift/swift/ray/base.py", line 168, in wrapper
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return func(self, *args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/yangpenghui1/ms-swift/swift/pipelines/train/sft.py", line 197, in run
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return self.train(trainer)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/yangpenghui1/ms-swift/swift/pipelines/train/sft.py", line 271, in train
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: trainer.train(resume_checkpoint)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/yangpenghui1/ms-swift/swift/trainers/mixin.py", line 903, in train
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: res = super().train(*args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/transformers/trainer.py", line 1424, in train
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return inner_training_loop(
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/transformers/trainer.py", line 1506, in _inner_training_loop
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: self._run_epoch(
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/transformers/trainer.py", line 1734, in _run_epoch
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/yangpenghui1/ms-swift/swift/rlhf_trainers/utils.py", line 613, in wrapper
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return func(self, *args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/yangpenghui1/ms-swift/swift/rlhf_trainers/gkd_trainer.py", line 1092, in training_step
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return HFSFTTrainer.training_step(self, model, inputs, num_items_in_batch)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/trl/trainer/sft_trainer.py", line 1338, in training_step
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return super().training_step(*args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/transformers/trainer.py", line 1934, in training_step
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: self.accelerator.backward(loss, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/accelerate/accelerator.py", line 2830, in backward
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: self.deepspeed_engine_wrapped.backward(loss, sync_gradients=self.sync_gradients, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/accelerate/utils/deepspeed.py", line 270, in backward
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: self.engine.backward(loss, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/utils/nvtx.py", line 33, in wrapped_fn
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ret_val = func(*args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/engine.py", line 2633, in backward
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: loss.backward(**backward_kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/_tensor.py", line 630, in backward
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: torch.autograd.backward(
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/autograd/init.py", line 364, in backward
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: _engine_run_backward(
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/autograd/graph.py", line 865, in _engine_run_backward
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/autograd/function.py", line 317, in apply
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return user_fn(self, *args)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/utils/checkpoint.py", line 325, in backward
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: torch.autograd.backward(outputs_with_grad, args_with_grad)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/autograd/init.py", line 364, in backward
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: _engine_run_backward(
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/autograd/graph.py", line 865, in _engine_run_backward
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/autograd/function.py", line 317, in apply
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return user_fn(self, *args)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 419, in backward
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ctx.pre_backward_function(ctx.module)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/utils/nvtx.py", line 33, in wrapped_fn
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ret_val = func(*args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 400, in _run_before_backward_function
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: self.pre_sub_module_backward_function(sub_module)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return func(*args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/zero/parameter_offload.py", line 509, in pre_sub_module_backward_function
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: param_coordinator.fetch_sub_module(sub_module, forward=False)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 1181, in _fn
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return fn(*args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/utils/nvtx.py", line 33, in wrapped_fn
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ret_val = func(*args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return func(*args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 325, in fetch_sub_module
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: self._fetch_sub_module_impl(current_submodule, forward, is_leaf)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 460, in _fetch_sub_module_impl
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: self.__all_gather_params(params_to_prefetch, forward)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/utils/nvtx.py", line 33, in wrapped_fn
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ret_val = func(*args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 520, in __all_gather_params
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: self._all_gather_params(nonquantized_params, forward, quantize=self.zero_quantized_weights)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/zero/partitioned_param_coordinator.py", line 549, in _all_gather_params
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: handle = param_group[0].all_gather_coalesced(param_group, quantize=quantize)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/utils/nvtx.py", line 33, in wrapped_fn
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ret_val = func(*args, **kwargs)
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 1491, in all_gather_coalesced
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: return _all_gather_coalesced(params, world_size, rank_in_group, use_secondary_tensor, ds_process_group,
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 1377, in _all_gather_coalesced
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: _all_gather_dtype(dtype_params[dtype], world_size, rank_in_group, ds_process_group, dtype))
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: File "/mnt/shared-storage-user/mllm/yangpenghui/miniconda3/envs/clawrl_swift/lib/python3.12/site-packages/deepspeed/runtime/zero/partition_parameters.py", line 1262, in _all_gather_dtype
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: [p.ds_tensor.to(get_accelerator().current_device_name()).to(allgather_dtype) for p in params],
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: torch.AcceleratorError: CUDA error: an illegal memory access was encountered
[2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: Search for cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. [2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. [2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: For debugging consider passing CUDA_LAUNCH_BLOCKING=1 [2026-06-21T01:26:03+08:00] simulate-qwen36-gpt-train-22558506-8beb9-cfcd2 >> [rank3]: Compile with TORCH_USE_CUDA_DSA` to enable device-side assertions.

How to Reproduce / 如何复现

ms-swift: 4.3.0
cuda: 12.8
GPU: H200

Additional Information / 补充信息

codex分析可能问题出在teacher侧，例如返回的top-K的index越界之类的

Start with swift/rlhf_trainers/gkd_trainer.py at training_step and swift/rlhf_trainers/utils.py at the wrapper, then trace the reported backward path through swift/pipelines/train/rlhf.py. Reproduce the intermittent GKD CUDA illegal-access failure and verify that the cause and a stable fix are covered by a repeatable test or run.