Revisit CRC options
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 108
- Forks
- 37
- PR merge metrics
- No merged PRs in 30d
Description
In the original SCR, we would compute CRCs against user files at various times. Those have been dropped in the switch to components. We'll want to either revive these or decide to drop them.
With SCR_CRC_ON_FLUSH SCR would compute and store CRC values when flushing files to the parallel file system. We then checked those CRCs when reading the files back during a fetch. This was to catch corruption to the file while on the file system or during our flush/fetch transfer. If SCR detected a mismatch, it marked the checkpoint as bad and wouldn't try to fetch it again in the future.
Note that sometimes users intentionally modify their checkpoint files before restarting their run, so this check needs to be optional. The user should also be able to mark a checkpoint as valid again in case SCR has already marked it as bad due to a previous failed CRC check.
With SCR_CRC_ON_COPY, SCR would compute the CRC on the original files at SCR_Complete_checkpoint/output. Those values were stored and then checked against the files after a rebuild to ensure the rebuild did not corrupt data.
With SCR_CRC_ON_DELETE SCR would compute the CRC against each file in cache before deleting it. This was more of a debugging check to ensure the file hadn't changed since SCR originally computed its CRC during the COPY. That could be used to detect a bug in SCR or a place where the application may have updated the file after calling SCR_Complete_checkpoint/output, which would be a bug how the application is using SCR.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing component-based checkpoint flow around SCR_Complete_checkpoint/output, flush, fetch, copy, rebuild, and delete operations. Review how SCR_CRC_ON_FLUSH, SCR_CRC_ON_COPY, and SCR_CRC_ON_DELETE worked in the original SCR, then determine whether CRC support should be restored or removed. Done criteria must include optional checks and a way to mark a CRC-failed checkpoint valid again.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100