llnl / llnl/UnifyFS

PnetCDF test leads to margo error, which leads to hang in ROMIO

Open
#783 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pnetcdf
Dominant language
C
Stars
122
Forks
34
PR merge metrics
No merged PRs in 30d

Description

While running a particular margo test

https://github.com/Parallel-NetCDF/PnetCDF/blob/master/test/largefile/high_dim_var.c

with 4 ranks on 2 nodes, a read from rank 2 invokes a failure on the server, which generates the following logs:
```
023-07-06T16:00:56 tid=872735 @ signal_new_requests() [unifyfs_request_manager.c:269] signaling new requests
2023-07-06T16:00:56 tid=873012 @ request_manager_thread() [unifyfs_request_manager.c:1802] RM[1511587981:1] got work
2023-07-06T16:00:56 tid=873012 @ rm_process_client_requests() [unifyfs_request_manager.c:1631] processing 1 client requests
2023-07-06T16:00:56 tid=873012 @ process_read_rpc() [unifyfs_request_manager.c:1324] processing mread[0] with 1 requests
2023-07-06T16:00:56 tid=873012 @ submit_read_request() [unifyfs_fops_rpc.c:252] handling read request (1 extents)
2023-07-06T16:00:56 tid=873012 @ pull_margo_bulk_buffer() [../../common/src/unifyfs_rpc_util.c:179] margo_bulk_transfer(buf_offset=0, len=1572864) failed
2023-07-06T16:00:56 tid=873012 @ pull_margo_bulk_buffer() [../../common/src/unifyfs_rpc_util.c:197] failed bulk transfer - transferred 0 of 1572864 bytes
2023-07-06T16:00:56 tid=873012 @ unifyfs_invoke_find_extents_rpc() [unifyfs_p2p_rpc.c:665] failed to get bulk chunk locations
2023-07-06T16:00:56 tid=873012 @ submit_read_request() [unifyfs_fops_rpc.c:279] failed to find extent locations
2023-07-06T16:00:56 tid=873012 @ process_read_rpc() [unifyfs_request_manager.c:1333] unifyfs_fops_read() failed
2023-07-06T16:00:56 tid=873012 @ rm_process_client_requests() [unifyfs_request_manager.c:1690] client rpc request 0 failed ("Mercury/Argobots operation error")
2023-07-06T16:00:56 tid=873012 @ request_manager_thread() [unifyfs_request_manager.c:1768] failed to process client rpc requests
```

The error code returned to the client for the read is 1004. That probably corresponds to one of these:

https://github.com/mercury-hpc/mercury/blob/55b95f72714bb0e4e0deeedf4fd78d116ea9476a/src/mercury_core_types.h#L102-L108

The read error happens during ``PMI_File_read_at_all`` which then leads to a deadlock in ROMIO:
https://github.com/pmodels/mpich/issues/6585

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the four-rank, two-node case using test/largefile/high_dim_var.c and trace the read through unifyfs_request_manager.c, unifyfs_fops_rpc.c, unifyfs_p2p_rpc.c, and common/src/unifyfs_rpc_util.c. Start with the failed bulk transfer and error 1004, then verify that PMI_File_read_at_all completes without the ROMIO deadlock described in MPICH issue 6585.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
distributed-systems, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.