llnl / llnl/UnifyFS

use file-based barrier for keyval fence and pids file

Open
#650 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
122
Forks
34
PR merge metrics
No merged PRs in 30d

Description

I see we still have a sleep(10) as our fence for the file system key/value store:

https://github.com/LLNL/UnifyFS/blob/a2d33f07af66886305d8740a754c9ff9a1895f7c/common/src/unifyfs_keyval.c#L800

To add a true barrier here, we could use a scheme where we use a write flock and have each server add a line (or a byte), then have all servers wait until that file reaches a certain line count or size with stat. For proper locking, this would likely require a file system like Lustre or GPFS.

Anyone know of problems with that approach?

Similarly, I think we could refactor creation of the "pids" file.

https://github.com/LLNL/UnifyFS/blob/a2d33f07af66886305d8740a754c9ff9a1895f7c/server/src/unifyfs_server.c#L404

I believe all this really requires is for rank 0 to create a file when it knows for sure that all servers have published their client-server rpc address, which we can do if we have a true barrier between servers from the key value store. The current sync logic waits for the file to hit a certain line count, but it doesn't need to know actual server pids:

https://github.com/LLNL/UnifyFS/blob/dev/util/unifyfs/src/unifyfs-rm.c#L221-L229

If we do need to collect pid values, that could also be done through the key value exchange. We could likely simplify that to just test for existence of the file at all.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the key/value fence in common/src/unifyfs_keyval.c around line 800, then inspect pids-file creation in server/src/unifyfs_server.c around line 404 and sync logic in util/unifyfs/src/unifyfs-rm.c around lines 221-229. Done would mean replacing the sleep-based coordination with a confirmed barrier and simplifying pids-file handling without losing server startup synchronization.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
distributed-systems, operating-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.