Restart bootstrapping
- Dominant language
- C
- Stars
- 108
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
In some applications, it is difficult to enable SCR separately in restart/checkpoint phases, meaning if they enable SCR for checkpoints, it is also enabled for restart. For those apps, it's not possible to turn SCR on in the middle of a ongoing simulation that has existing checkpoints, which were not written through SCR.
Perhaps we could add a "bootstrap" mode in which the SCR restart API acts like it has loaded a restart, but doesn't really.
To use this, the app would:
- set SCR_RESTART_BOOTSTRAP=1 to turn on boostrap mode.
- set SCR_CURRENT with the name of the checkpoint, as they would have named it if they had written it using SCR
When in bootstrap mode, then:
- SCR_Init would throw an error if SCR_CURRENT is not set. If it is set, disable fetch (and purge the cache?), and internally mark have_restart=1, enable bypass mode? It could disable bootstrap if an index file exists in order to protect user from bootstrapping again after a checkpoint is written?
- SCR_Have_restart would return 1 and the name that was given in SCR_CURRENT
- SCR_Start_restart would return the name that was given in SCR_CURRENT
- SCR_Route_file would copy the input string verbatim into the output buffer (or still prepend the CWD on relative paths, and check absolute path is in PREFIX?)
- SCR_Complete_restart would compute an allreduce(valid) and return the result. If unsuccessful, it would not update the index file, it would not attempt to load another checkpoint, and it would set have_restart = 0. If successful, set internal counters to max in PREFIX directory (to avoid clobbering existing files in case user left bootstrap mode on in a subsequent run by accident).
This would let an app call the SCR restart API and get the same behavior as though SCR had known about and loaded the checkpoint. They would likely only do this in one run and turn it off after SCR writes its first checkpoint.
Contributor guide
Research direction
Start by tracing the SCR_Init, SCR_Have_restart, SCR_Start_restart, SCR_Route_file, and SCR_Complete_restart APIs and the handling of SCR_RESTART_BOOTSTRAP and SCR_CURRENT. Define the bootstrap behavior for initialization, restart routing, completion success or failure, index updates, and counter handling, including the unresolved fetch, cache, and path questions. Done means the restart API behaves as specified for an existing non-SCR checkpoint without clobbering later SCR files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100