More reliable server bootstrapping at scale
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 122
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
### System information
Summit
### Describe the problem you're observing
The `unifyfsd` bootstrapping (i.e., establishing connections between peers) often fails when a large number of compute nodes are used, e.g., 500+ servers.
`unifyfsd` relies on `pmi2` or `pmix` to acquire job allocation information and group services like fence/barrier. However, these options seem unreliable when a job is launched with a large number of nodes (at least on summit as I have experienced). As a last resort, `unifyfsd` can bootstrap with peers without help from external libraries, by using a shared file system. However, there are also times that the shared file system (like GPFS) behaves badly (e.g., sudden slowdown of operations). I think we need to investigate if we can make the bootstrapping process more reliable.
### Describe how to reproduce the problem
Run the unifyfsd with a large number of servers, e.g., 500+ servers.
### Include any warning or errors or releveant debugging data
N/A
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the unifyfsd bootstrapping path and trace how PMI2, PMIx, and shared-file-system fallback establish peer connections. Reproduce the failure with 500+ servers on Summit and compare behavior across the available bootstrap methods. Done means peer connections are established reliably at that scale, including when the shared filesystem is slow.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- distributed-systems, hpc
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100