Multiple servers start up and execute as independent single-task jobs
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 122
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
Hit a corner case that took a while to debug. Since I like to run under totalview, I still prefer to launch the servers by hand. That looks something like this hacky thing:
```
procs=$SLURM_NNODES
srun -n $procs -N $procs touch /var/tmp/unifyfs.conf
srun -n $procs -N $procs mkdir /dev/shm/unifyfs
export UNIFYFS_CONFIGFILE=/var/tmp/unifyfs.conf
export UNIFYFS_SERVER_LOCAL_EXTENTS=0
export UNIFYFS_SHAREDFS_DIR=/p/lustre2/user1
export UNIFYFS_DAEMONIZE=off
totalview srun -a -n $procs -N $procs `pwd`/bin/unifyfsd
```
However, in this case, I apparently built without PMI support, and I had forgotten to create a servers hostfile. In that situation each server initializes ```glb_pmi_rank = 0``` and ```glb_pmi_size = 1```. During the run, each server thinks it is server rank 0, and so it generates key/value pairs using that information.
Then I ran a job with two nodes, one server per node, and one client per node. Each client wrote one extent, and the extents do not overlap. Each server inserted one key/value pair into MDHIM while both assumed they were server rank=0. When reading the data back out, a server ends up reading back both keys from MDHIM successfully, but then it thinks both extents are local, since ```delegator_rank=0``` in both key/value pairs. This results in a file that is the correct size, but the read returns "corrupted" data.
Not sure what to do about it yet, but I wanted to document the case to capture the details. Maybe we can make the server host file be a required setting when PMI isn't being used?
Also, what happens if PMI is enabled but then one sets a hostfile?
I did verify things worked again when I created the servers hostfile, which I did with this other hack:
```
export UNIFYFS_SERVER_HOSTFILE=/p/lustre2/user1/unifyfs_server_hosts
rm -f $UNIFYFS_SERVER_HOSTFILE
echo $SLURM_NNODES > $UNIFYFS_SERVER_HOSTFILE
srun -n $SLURM_NNODES -N $SLURM_NNODES /bin/hostname >> $UNIFYFS_SERVER_HOSTFILE
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the bin/unifyfsd launch path and inspect how PMI state and UNIFYFS_SERVER_HOSTFILE are handled. Reproduce the two-node, one-server-per-node case with PMI absent, then compare it with the hostfile setup; done means server ranks and delegator ranks are distinct and the resulting extents are read correctly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100