dfind slurm job proccess end too far apart
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 200
- Forks
- 85
- Avg merge
- 3d 21h
- Merged PRs (30d)
- 2
Description
When running a modified dfind that printed out extra data (uid, gid, blocks, size) on a whole file system czlustre3 (680 million items), the processes ended far enough apart that slurm killed the remaining processes. One option is to just tell slurm to allow a larger difference in process end times, but I think this issue is already solved for dwalk, such that dwalk can do a whole file system and do a --text printout and the processes all end at approximately the same time.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing dfind's process completion behavior with dwalk's whole-filesystem --text path, then investigate how the differing completion times interact with Slurm's process-end tolerance. Reproduce the issue on a large filesystem and define done as having all processes finish close enough together that Slurm does not kill the remaining work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, hpc
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100