llnl / llnl/UnifyFS

Issues/bugs observed when running real applications at large scales

Open
#812 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
122
Forks
34
PR merge metrics
No merged PRs in 30d

Description

Issues/bugs I noticed while running large-scale real applications on Frontier.
I use this for bookkeeping purpose. I will create a PR to fix them in the future.
The issues reported here occur only at large scales, e.g., 628-node FLASH-X runs.

### System information

The issues are not system dependent.

### Describe the problem you're observing

Most of the issues in the end will lead to Mercury TIMEOUT errors. Then the I/O (e.g., HDF5) will fail.

#### 1:

This is inside the `unifyfs_invoke_filesize_rpc()` function. So the rpc id should be `filesize_id` not `metaget_id`.
https://github.com/LLNL/UnifyFS/blob/58ece4441716678f5111a6dbff9baadd6188c2b6/server/src/unifyfs_p2p_rpc.c#L981

The bug causes that filesize rpc calls are never handled, all waiting forever.
We need to carefully examine if we have similar bugs like this. Best to have unit tests to cover all RPC routines.

#### 2:
During servers initialization process, server rank 0 acts as coordinator and performs a tree-based broadcast.
The hard-coded 5 secs timeout may not be enough for a large number of servers. I have to increase it a little to avoid the timeout error for 628-node Flash runs.
https://github.com/LLNL/UnifyFS/blob/58ece4441716678f5111a6dbff9baadd6188c2b6/server/src/unifyfs_group_rpc.c#L967

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with server/src/unifyfs_p2p_rpc.c at unifyfs_invoke_filesize_rpc() and server/src/unifyfs_group_rpc.c at the initialization broadcast timeout. Review the referenced RPC identifiers and hard-coded timeout, then inspect nearby RPC routines for similar issues. Done means the reported large-scale failures are addressed and unit-test coverage is added for the relevant RPC routines.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
distributed-systems, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.