llnl / llnl/scr

SCR in SPIO

Open
#229 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
108
Forks
37
PR merge metrics
No merged PRs in 30d

Description

Currently, SPIO has a light-weight integration with SCR, in which the IOManager only calls SCR_Route_file for the user. All other SCR calls are made from the application. For example, the application code might look like:
```
std::string name = "time." + std::to_string(t);
SCR_Start_output(name.c_str(), SCR_FLAG_CHECKPOINT);

// IOManager.write() internally calls SCR_Route_file
IOManager writer(comm, true);
writer.write(ds->getRoot(), num_files, ckpt_path, "sidre_hdf5");

int valid = 1;
int complete_rc = SCR_Complete_output(valid);
```

This light-weight integration provides flexibility:
- User has direct control of the SCR dataset name and flags arguments passed to SCR_Start_output
- User can pass a subcommunicator to IOManager, i.e., comm does not need to be MPI_COMM_WORLD
- User can specify the number of underlying files that IOManager will write
- User can write more than one dataset, for example perhaps a checkpoint consists of two datasets, i.e., they could make more than one call to writer.write as part of a single SCR dataset if desired

Many users could benefit from a deeper SCR integration in which the IOManager also calls Start_output and Complete_output for them. For this, SPIO will need to infer parameter values for SCR calls, like the dataset name and the checkpoint/output flags, or its interface will need to be extended to take those are arguments from the user. For example:
```
std::string scr_name = "time." + std::to_string(t);
int scr_flags = SCR_FLAG_CHECKPOINT;

// IOManager.write() calls SCR_Start_otuput(scr_name, scr_flags), SCR_Route_file, and SCR_Complete_output
IOManager writer(comm, true);
writer.write(ds->getRoot(), num_files, ckpt_path, scr_name, scr_flags, "sidre_hdf5");
```

With this extended interface, we keep:
- User has direct control of the SCR dataset name and flags arguments
- User can specify the number of underlying files that IOManager will write

Though some differences:
- User must now call writer.write() from all procs in MPI_COMM_WORLD (though I think perhaps the comm passed to the IOManager constructor could still be a subcommunicator, so that only a subset of procs would write out a datastore)
- A process can only submit one datastore to an SCR dataset

The read direction would be similar, except the SPIO interface would need to return a flag indicating whether the restart succeeded and it should return the checkpoint name in an output parameter to the user. We need to figure out if SCR_Have_restart should also be wrapped.

TODO: One tricky part is to figure out how to take the checkpoint name returned in SCR_Start_restart and map to the file name for SCR_Route_file.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing IOManager.write() path and its SCR_Route_file integration, then review the SCR_Start_output, SCR_Complete_output, SCR_Start_restart, and SCR_Have_restart entry points mentioned in the issue. Define the write and read interfaces, including checkpoint-name-to-file mapping, and validate that the resulting behavior preserves the stated communicator and dataset requirements.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.