llnl / llnl/UnifyFS

More reliable server bootstrapping at scale

Open
#568 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
122
Forks
34
PR merge metrics
No merged PRs in 30d

Description

### System information
Summit

### Describe the problem you're observing
The `unifyfsd` bootstrapping (i.e., establishing connections between peers) often fails when a large number of compute nodes are used, e.g., 500+ servers.

`unifyfsd` relies on `pmi2` or `pmix` to acquire job allocation information and group services like fence/barrier. However, these options seem unreliable when a job is launched with a large number of nodes (at least on summit as I have experienced). As a last resort, `unifyfsd` can bootstrap with peers without help from external libraries, by using a shared file system. However, there are also times that the shared file system (like GPFS) behaves badly (e.g., sudden slowdown of operations). I think we need to investigate if we can make the bootstrapping process more reliable.

### Describe how to reproduce the problem
Run the unifyfsd with a large number of servers, e.g., 500+ servers.

### Include any warning or errors or releveant debugging data

N/A

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the unifyfsd bootstrapping path and trace how PMI2, PMIx, and shared-file-system fallback establish peer connections. Reproduce the failure with 500+ servers on Summit and compare behavior across the available bootstrap methods. Done means peer connections are established reliably at that scale, including when the shared filesystem is slow.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
distributed-systems, hpc
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.