Is it possible to avoid loading the same checkpoint into RAM / GPU?

When training on multiple GPUs the script loads the same checkpoint into system RAM / GPU. Would it be possible to avoid this?

Start by locating the multi-GPU training entry point and the code that loads checkpoints into system RAM or GPU memory. Reproduce the behavior with multiple GPUs, then determine whether redundant checkpoint loads can be avoided without changing training behavior; done means the checkpoint is not unnecessarily loaded per GPU.

Written by the indexing model from the issue text.