Restart with a different number of ranks
@mcfadden8 is already working on this.
Since Mar 17, 2025.
- Dominant language
- C
- Stars
- 108
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
SCR currently allows an application to restart with a different number of ranks. However, one cannot call the SCR restart API in that case.
https://scr.readthedocs.io/en/latest/users/integration.html#restart-without-scr
This is awkward for applications that can otherwise use the SCR restart API when restarting with the same number of ranks, since they then need to have two code paths:
1) if restarting with same number of ranks --> use SCR restart API
2) if restarting with different number of ranks --> do not use the SCR restart API
It would be nice to merge these. It should be possible when leaving the files on the parallel file system, but there are checks and logic in the fetch process that currently do not support it.
One known problem is in reading the rank2file map. This scatters the files using kvtree, and it currently requires the exact same number of ranks to read the file which wrote it.
https://github.com/LLNL/scr/blob/79ff7ed65993331f2c4948e4a76eacb27c3d14ba/src/scr_fetch.c#L169
We could work around that to distribute the file info to the ranks in the current run. We could just have kvtree decide how the info gets spread out, or we'd need to modify the kvtree API so that the calling ranks can specify the new mapping.
For the remainder of the function, we ``stat`` each file to verify that it exists. It would be nice to keep that, and it's easy to handle.
https://github.com/LLNL/scr/blob/79ff7ed65993331f2c4948e4a76eacb27c3d14ba/src/scr_fetch.c#L251-L258
The trickier part is that we then fill in the local filemap data structure with info about each file that a rank "owns". It's not clear what to do in this case. One option would be to have each rank register every file as though all files are shared by all ranks. This is not exactly scalable, but perhaps it's the safest option, since we don't know how they will be accessed.
https://github.com/LLNL/scr/blob/79ff7ed65993331f2c4948e4a76eacb27c3d14ba/src/scr_fetch.c#L267-L272
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.