NVIDIA-NeMo / NVIDIA-NeMo/RL

gemma-3-4b-it got nan probs_ratio in both FSDP1/FSDP2

Open
#419 5 comments 0 reactions 1 assignee Claimed by @yfw View on GitHub
bug t-pytdensor
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

repro:
commit id: `0d05ba2d493907a24fc48e949da1cda192967907`
```
RUN_COMMAND="uv run python examples/run_grpo_math.py \
--config examples/configs/grpo_math_8B.yaml \
grpo.max_num_steps=100 \
grpo.val_period=1000 \
policy.model_name=google/gemma-3-4b-it \
policy.dtensor_cfg.enabled=true \
policy.dtensor_cfg.tensor_parallel_size=2 \
policy.generation.vllm_cfg.tensor_parallel_size=2 \
checkpointing.enabled=false \
logger.wandb_enabled=true \
logger.tensorboard_enabled=false \
logger.monitor_gpus=true \
logger.wandb.project={WANDB_PROJECT} \
logger.wandb.name=${EXP_NAME} \
cluster.num_nodes=${NUM_ACTOR_NODES} \
cluster.gpus_per_node=8"
```

at step15, some metrics are still normal, but the `probs_ratio` is nan.
at step16, almost all the metrics become abnormal.
![Image](https://github.com/user-attachments/assets/5d1f2d12-a64e-445e-8940-7d62f2f13a67)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.