Unexpected result nan？

我正常训练过程中发生nan情况，数据没有问题，如何解决呢（我测试了不同版本的sft模型：pf32，bf16等）？（重新启动resume，仍然在step205左右nan）
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) [2025-10-11 11:16:52] [decorator.py (306)] [ERROR] [actor_train-1 1 / 8][PID 18175] Traceback (most recent call last):
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/checkpoint/binary/train_package/roll/distributed/scheduler/decorator.py", line 295, in inner
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) result = func(*args, **kwargs)
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/checkpoint/binary/train_package/roll/pipeline/base_worker.py", line 104, in train_step
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) pg_metrics = self.strategy.train_step(batch=data, loss_func=self.loss_func)
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/checkpoint/binary/train_package/roll/distributed/strategy/megatron_strategy.py", line 454, in train_step
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) metrics_tensors: List[Dict[str, "torch.Tensor"]] = self.forward_backward_func(
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/root/.local/lib/python3.10/site-packages/megatron/core/pipeline_parallel/schedules.py", line 535, in forward_backward_no_pipelining
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) backward_step(input_tensor, output_tensor, output_tensor_grad, model_type, config)
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/root/.local/lib/python3.10/site-packages/megatron/core/pipeline_parallel/schedules.py", line 409, in backward_step
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) custom_backward(output_tensor[0], output_tensor_grad[0])
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/root/.local/lib/python3.10/site-packages/megatron/core/pipeline_parallel/schedules.py", line 160, in custom_backward
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) Variable._execution_engine.run_backward(
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/autograd/function.py", line 307, in apply
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) return user_fn(self, *args)
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/root/.local/lib/python3.10/site-packages/megatron/core/tensor_parallel/random.py", line 469, in backward
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) torch.autograd.backward(outputs, args)
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) _engine_run_backward(
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/opt/conda/envs/python3.10.13/lib/python3.10/site-packages/torch/autograd/graph.py", line 823, in _engine_run_backward
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/root/.local/lib/python3.10/site-packages/megatron/core/distributed/distributed_data_parallel.py", line 510, in hook
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) self.param_to_bucket_group[param].register_grad_ready(param)
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/root/.local/lib/python3.10/site-packages/megatron/core/distributed/param_and_grad_buffer.py", line 450, in register_grad_ready
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) self.start_grad_sync()
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/root/.local/lib/python3.10/site-packages/megatron/core/distributed/param_and_grad_buffer.py", line 305, in start_grad_sync
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) self.check_grads(
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/root/.local/lib/python3.10/site-packages/megatron/core/distributed/param_and_grad_buffer.py", line 177, in check_grads
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) rerun_state_machine.validate_result(
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) File "/root/.local/lib/python3.10/site-packages/megatron/core/rerun_state_machine.py", line 505, in validate_result
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) raise RuntimeError(full_message)
(ActorWorker(actor_train-1) pid=18175, ip=33.199.184.82) RuntimeError: Rank 1, node c68f01266.ea119, device 0, iteration -1: Unexpected result nan (message='found NaN in local grad norm for bucket #0 in backward pass before data-parallel communication collective')

Start at pipeline/base_worker.py:104 and distributed/strategy/megatron_strategy.py:454, then follow the traceback into Megatron's backward pass and gradient validation. Reproduce the run around step 205 and determine where the local gradient norm first becomes NaN; done means identifying a reproducible cause and confirming training no longer fails there.

Written by the indexing model from the issue text.