allenai / allenai/OLMo-core

Free staged state dict during async checkpointer callbacks

Offen
#856 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
1.5k
Forks
315
Ø Merge
1 T. 9 Std.
Gemergte PRs (30 T.)
11

Beschreibung

When using `save_async=True` in a `CheckpointerCallback`, the state dict is staged to RAM and passed to [`Checkpointer.save_async`](https://github.com/allenai/OLMo-core/blob/v2.6.0/src/olmo_core/train/trainer.py#L1015). That whole state dict is kept in memory for the duration of callbacks as well, even though the callbacks have no access to it. This is somewhat non-obvious. It happens because that state dict ultimately gets [submitted to a `ThreadPoolExecutor`](https://github.com/pytorch/pytorch/blob/v2.14.0/torch/distributed/checkpoint/_async_process_executor.py#L444) which stores it in the `args` tuple of a work item, and that work item sticks around until the future *and all its callbacks* are finished.

The state dict is just a normal `dict`, so we could call `dict.clear()` on it at the end of the checkpoint, and that should free the memory before starting the callbacks.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.