[Bug] save_model fires unconditionally at the final step regardless of --save-interval; final optimizer-state write can kill the job

Summary

save_model fires unconditionally at the final training step regardless of --save-interval, because should_run_periodic_action returns True at the last step. Users who set --save-interval larger than the total number of rollout steps (intending "no checkpointing") still get a full optimizer-state checkpoint written at the end — and on our setups this final write killed the training job twice on two different machines.

Environment

• slime: main @ 4c193f1f (also reproduced on an older checkout, different machine)
• 2× separate vast.ai H100 boxes, single-GPU colocate, ray 2.58, sglang (slime-pinned)
• GRPO run, 50 rollout steps, --save /workspace/ckpt --save-interval 200 (interval > horizon = "no saves intended")

Observed

1. At rollout_id == num_rollout - 1, train.py:70-79 calls actor_model.save_model(...) because should_run_periodic_action(rollout_id, args.save_interval, ...) returns True at the final step (slime/utils/misc.py). The --save-interval 200 setting is silently overridden.
2. The final save writes a ~36G Megatron checkpoint including optimizer state. On two independent machines the actor died inside this save:
   • Box A: torch distributed-checkpoint write failed with unexpected pos 704 vs 598 (corrupt write), job hung until manually stopped.
   • Box B: actor vanished mid-save — driver raised ray.exceptions.ActorUnavailableError: ... RPC error: Socket closed rpc_code: 14 at slime/ray/actor_group.py:153. Because the crash is in save_model before the final-step eval (train.py:95-96), the in-job eval never runs and the whole run's rollouts are lost.
3. Workaround confirmed: --no-save-optim shrinks the final checkpoint to ~6G weights-only and the save completes (we never resume from GRPO checkpoints, so optimizer state was pure liability).

Expected behavior

One of:

• --save-interval > num_rollouts should mean "no saves at all" (including the final step), or
• an explicit --no-final-save flag, or
• at minimum a docs note that the final step always saves regardless of --save-interval.

Related

• #2290 / #1914 / #1798 (checkpoint retention bounding — same underlying pain: full optimizer-state checkpoints are heavy and dangerous)

Fix that worked for us

--no-save-optim

(pre-existing flag, slime/utils/arguments.py:872)

Start in train.py:70-79 and inspect should_run_periodic_action in slime/utils/misc.py to trace why the final rollout triggers save_model despite --save-interval 200. Review the existing --no-save-optim argument in slime/utils/arguments.py:872 and the reported GRPO reproduction; done means the chosen final-save behavior is explicit and the unintended optimizer-state write no longer occurs.