[REQUEST] Add an option to decide whether to store the checkpoint and rng_state.

Recently I've been using deepspeed and transformers simultaneously, and when I set the save_strategy to epoch (or step), it will store the checkpoint (xx.pt), and rng_state automatically. Also, when I set stage3_gather_fp16_weights_on_model_save=true , it will store the pytorch_model.bin. These satisfy our needs in most of the situations.

However, if we train a LLM like llama, the size of the checkpoint on one single GPU can reach 7.5G. So it will be nearly 60G if we use 8 GPUs to train. It has such a high demand for disk space especially when a whole team is using the same server.

I read the source code in trainer.train function in transformers and found that when enabling deepspeed, it automatically stores checkpoint and rng_state, but as I mentioned above, can we choose not to store them and just leave the .bin file? I think we may need a parameter option to decide this.

One solution is that we can divide the save_strategy into smaller parts like save_checkpoint_strategy and save_rng_state_strategy, and when we set them to no, they will not be stored. I think this could be done by changing some if-else code logic in the source code.

Best wishes.

Start with the trainer.train entry point in Transformers and trace the DeepSpeed save handling it invokes. Follow where the checkpoint, rng_state, and pytorch_model.bin files are written, along with the existing save controls. Done means users can choose whether checkpoint and RNG-state files are stored without preventing the requested model file from being saved.