llnl / llnl/scr

SCR compression

Open
#205 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
108
Forks
37
PR merge metrics
No merged PRs in 30d

Description

We should consider adding compression in SCR. We mention wanting to do it in https://computing.llnl.gov/projects/scalable-checkpoint-restart-for-mpi and in src/scr_io.c:

```C
/* TODO: could perhaps use O_DIRECT here as an optimization */
/* TODO: could apply compression/decompression here */
/* copy src_file (full path) to dest_path and return new full path in dest_file */
int scr_file_copy(
const char* src_file,
const char* dst_file,
unsigned long buf_size,
uLong* crc)
{
...
```
I can see cases where it would be beneficial, and cases where it wouldn't. If we did it, I'd recommend we use zstandard (https://github.com/facebook/zstd) which is currently the best compressor/decompressor out there.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading src/scr_io.c, especially scr_file_copy, and the linked SCR project description to understand where checkpoint data is copied. Define when compression should apply and how zstandard would fit before implementing it; done requires an agreed design and validation of both beneficial and non-beneficial cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.