llnl / llnl/UnifyFS

Multiple servers start up and execute as independent single-task jobs

Open
#531 20 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
122
Forks
34
PR merge metrics
No merged PRs in 30d

Description

Hit a corner case that took a while to debug. Since I like to run under totalview, I still prefer to launch the servers by hand. That looks something like this hacky thing:
```
procs=$SLURM_NNODES
srun -n $procs -N $procs touch /var/tmp/unifyfs.conf
srun -n $procs -N $procs mkdir /dev/shm/unifyfs

export UNIFYFS_CONFIGFILE=/var/tmp/unifyfs.conf
export UNIFYFS_SERVER_LOCAL_EXTENTS=0
export UNIFYFS_SHAREDFS_DIR=/p/lustre2/user1
export UNIFYFS_DAEMONIZE=off

totalview srun -a -n $procs -N $procs `pwd`/bin/unifyfsd
```
However, in this case, I apparently built without PMI support, and I had forgotten to create a servers hostfile. In that situation each server initializes ```glb_pmi_rank = 0``` and ```glb_pmi_size = 1```. During the run, each server thinks it is server rank 0, and so it generates key/value pairs using that information.

Then I ran a job with two nodes, one server per node, and one client per node. Each client wrote one extent, and the extents do not overlap. Each server inserted one key/value pair into MDHIM while both assumed they were server rank=0. When reading the data back out, a server ends up reading back both keys from MDHIM successfully, but then it thinks both extents are local, since ```delegator_rank=0``` in both key/value pairs. This results in a file that is the correct size, but the read returns "corrupted" data.

Not sure what to do about it yet, but I wanted to document the case to capture the details. Maybe we can make the server host file be a required setting when PMI isn't being used?

Also, what happens if PMI is enabled but then one sets a hostfile?

I did verify things worked again when I created the servers hostfile, which I did with this other hack:
```
export UNIFYFS_SERVER_HOSTFILE=/p/lustre2/user1/unifyfs_server_hosts
rm -f $UNIFYFS_SERVER_HOSTFILE
echo $SLURM_NNODES > $UNIFYFS_SERVER_HOSTFILE
srun -n $SLURM_NNODES -N $SLURM_NNODES /bin/hostname >> $UNIFYFS_SERVER_HOSTFILE
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the bin/unifyfsd launch path and inspect how PMI state and UNIFYFS_SERVER_HOSTFILE are handled. Reproduce the two-node, one-server-per-node case with PMI absent, then compare it with the hostfile setup; done means server ranks and delegator ranks are distinct and the resulting extents are read correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.