Training stalls after several hours with no noticeable error

**Describe the bug**

I keep running into an issue where NeMo-RL randomly stalls after several hours of runtime and can’t pin down the cause yet. I don’t see any errors or other signs that something went wrong, but progress will suddenly halt and GPU usage goes down to zero (with GPU memory still being high and W&B considering the application to still be running). AFAICT, the cluster is still healthy and I’ve tried on a few different clusters with both single and two-node configs.

**Steps/Code to reproduce bug**

Setup a one-node Ray cluster on 8x H100 GPUs using the `nvcr.io/nvidia/nemo-rl:v0.4.0.nemotron_3_nano` container.

Launch an RL job on the cluster with `uv run examples/nemo_gym/run_grpo_nemo_gym.py --config examples/nemo_gym/grpo_qwen3_4b.yaml` (config to be provided in comments).

Let the job run for greater than 4 hours. In multiple tests, the training will randomly freeze sometime between 4 and 6 hours and will be stopped indefinitely.

**Expected behavior**

I would expect the training process to run indefinitely and/or a meaningful error message be displayed if something went wrong to stall the training process.

**Additional context**

Config file to be added in the comments.

Note that the `checkpoint_must_save_by` value is set to `null` so it shouldn't be related to early termination of the training for the checkpoint.