Checksumming a list of files in parallel
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 200
- Forks
- 85
- Avg merge
- 3d 21h
- Merged PRs (30d)
- 2
Description
Hi - is there a way I'm not thinking of to use mpifileutils tools to md5sum a list of files in parallel. I recently ended up needing to use xargs to do checksum a list of ~50 files on a single compute node, but it would have been nicer to use something like dparallel to checksum that list across multiple nodes. There's not much dparallel documentation, but from what I see I'm thinking dparallel is intended for doing the same thing in parallel on several systems as opposed to doing the same task to a list of files that gets split up. Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue does not name any files, tests, or entry points. Start by reading the dparallel documentation and checking how its work is distributed across systems; compare that behavior with the requested splitting of a file list across compute nodes. Done would require a clearly defined supported workflow or feature for parallel checksumming of the list.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, hpc
- Domain
- distributed-systems, hpc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100