CT Tool for Ariane-133: Snapshot generation halt

Hello, I wanted to ask about the Ariane-133 design (from MacroPlacement). I ran the CT tool for 11 days (Intel Xeon, 132GB RAM, and I used 3 collect jobs, no GPUs), however it seemed that the tool stopped generating snapshots after day two. On Tensorboard, I also noticed that the losses plateaued around day two. Is there a reason for this? Additionally, is there a measure in place to know when the tool is done? It seems like it finished generating snapshots but continued to run. Should it stop at a certain point? Lastly, I am curious as to why the checkpoints directory was empty (no checkpoints created, even after 31k steps).

Thank you so much!

snapshot results taken on 01/26/25:

Tensorboard Results:
plateau occurs after 1.782 days:

all jobs ended manually on day 11:





Start by reproducing the Ariane-133 run with the CT tool and reviewing the TensorBoard output, snapshot results, and checkpoints directory described in the report. Trace snapshot generation, checkpoint creation, and the run's stopping condition. Done means explaining the plateau and missing checkpoints, and documenting or fixing how completion is detected.

Written by the indexing model from the issue text.