Issues/bugs observed when running real applications at large scales
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 122
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
Issues/bugs I noticed while running large-scale real applications on Frontier.
I use this for bookkeeping purpose. I will create a PR to fix them in the future.
The issues reported here occur only at large scales, e.g., 628-node FLASH-X runs.
### System information
The issues are not system dependent.
### Describe the problem you're observing
Most of the issues in the end will lead to Mercury TIMEOUT errors. Then the I/O (e.g., HDF5) will fail.
#### 1:
This is inside the `unifyfs_invoke_filesize_rpc()` function. So the rpc id should be `filesize_id` not `metaget_id`.
https://github.com/LLNL/UnifyFS/blob/58ece4441716678f5111a6dbff9baadd6188c2b6/server/src/unifyfs_p2p_rpc.c#L981
The bug causes that filesize rpc calls are never handled, all waiting forever.
We need to carefully examine if we have similar bugs like this. Best to have unit tests to cover all RPC routines.
#### 2:
During servers initialization process, server rank 0 acts as coordinator and performs a tree-based broadcast.
The hard-coded 5 secs timeout may not be enough for a large number of servers. I have to increase it a little to avoid the timeout error for 628-node Flash runs.
https://github.com/LLNL/UnifyFS/blob/58ece4441716678f5111a6dbff9baadd6188c2b6/server/src/unifyfs_group_rpc.c#L967
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with server/src/unifyfs_p2p_rpc.c at unifyfs_invoke_filesize_rpc() and server/src/unifyfs_group_rpc.c at the initialization broadcast timeout. Review the referenced RPC identifiers and hard-coded timeout, then inspect nearby RPC routines for similar issues. Done means the reported large-scale failures are addressed and unit-test coverage is added for the relevant RPC routines.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- distributed-systems, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100