[Feature Request] Documentation explaining the recommended way to save and load agents

Motivation

I would like a concrete documented example of how to save RL agents after they have been trained, and also how to load them afterwards to resume training. In my current projects, I always switch between different methods and I have not yet found an approach that feels "right". For example,

• Should only the nn.Modules be saved or the TensorDictModules?
• Should the loss module be saved?
• Saving the state_dict is usually recommended for nn.Modules. Is it the same way here?

I know that https://docs.pytorch.org/rl/stable/tutorials/export.html already explains how to export only the policy for inference, but I'm looking for something more general which would allow to resume training of the agent.

#2620 and #1089 are related.

Solution

I'm proposing that one of the following pages gets added to the documentation:

1. A separate page that describes how to save a DQN agent (or some other off-policy method). Network weights, optimizer state and the replay buffer should be saved so that training can resume.
2. In the PPO tutorial (https://docs.pytorch.org/rl/stable/tutorials/coding_ppo.html#), add a section at the end that explains how to save the trained agent. The drawback with this approach is that it doesn't show how to save a replay buffer.

Alternatives

I don't think there is. Saving models after they have been trained is an essential part of the RL workflow and the recommended way to do this should definitely be documented somewhere.

Checklist

• I have checked that there is no similar issue in the repo (required)