Clarification on client host memory requirement for 1T model checkpointing – changed between v2.0 and v3.0?

Hi MLCommons Storage team,

I have a question regarding the client host memory requirement for the checkpointing benchmark, specifically for the 1T model.

According to the current README (https://github.com/mlcommons/storage/tree/main/checkpointing), Table 2 shows that the checkpoint size for the 1T model is 18 TB. The README also states:

"Note: the aggregate simulated accelerator memory across all nodes must be sufficient to accommodate the model’s checkpoint size."

Based on this, I previously understood that the total client host memory across all nodes must be at least 18 TB to run the 1T checkpointing benchmark.

However, when I actually ran the benchmark this year (presumably with the MLPerf Storage v3.0 codebase), I found that the benchmark ran successfully even though the total client host memory was less than 18 TB. This was not possible with the v2.0 codebase, where the benchmark would fail if the memory was insufficient.

This leads me to believe that there may have been a change between v2.0 and v3.0 regarding how the client host memory requirement is enforced. Could you please clarify:

Has the memory requirement been relaxed or removed in v3.0?

If so, what is the new rule or validation logic for client host memory?

Is the --client-host-memory-in-gb parameter still strictly validated against the checkpoint size, or is it now only used for cache clearing decisions (e.g., the "3× memory" rule mentioned in the docs)?

I would appreciate any clarification on this. Thank you!

Start with the checkpointing README, especially Table 2 and the note about aggregate simulated accelerator memory, then search the v2.0 and v3.0 code for --client-host-memory-in-gb and its validation. Compare the relevant behavior and document whether the requirement changed, including the cache-clearing rule and the conditions that define completion.