Free staged state dict during async checkpointer callbacks
- Vorherrschende Sprache
- Python
- Sterne
- 1.5k
- Forks
- 315
- Ø Merge
- 1 T. 9 Std.
- Gemergte PRs (30 T.)
- 11
Beschreibung
When using `save_async=True` in a `CheckpointerCallback`, the state dict is staged to RAM and passed to [`Checkpointer.save_async`](https://github.com/allenai/OLMo-core/blob/v2.6.0/src/olmo_core/train/trainer.py#L1015). That whole state dict is kept in memory for the duration of callbacks as well, even though the callbacks have no access to it. This is somewhat non-obvious. It happens because that state dict ultimately gets [submitted to a `ThreadPoolExecutor`](https://github.com/pytorch/pytorch/blob/v2.14.0/torch/distributed/checkpoint/_async_process_executor.py#L444) which stores it in the `args` tuple of a work item, and that work item sticks around until the future *and all its callbacks* are finished.
The state dict is just a normal `dict`, so we could call `dict.clear()` on it at the end of the checkpoint, and that should free the memory before starting the callbacks.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.