PnetCDF test leads to margo error, which leads to hang in ROMIO
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 122
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
While running a particular margo test
https://github.com/Parallel-NetCDF/PnetCDF/blob/master/test/largefile/high_dim_var.c
with 4 ranks on 2 nodes, a read from rank 2 invokes a failure on the server, which generates the following logs:
```
023-07-06T16:00:56 tid=872735 @ signal_new_requests() [unifyfs_request_manager.c:269] signaling new requests
2023-07-06T16:00:56 tid=873012 @ request_manager_thread() [unifyfs_request_manager.c:1802] RM[1511587981:1] got work
2023-07-06T16:00:56 tid=873012 @ rm_process_client_requests() [unifyfs_request_manager.c:1631] processing 1 client requests
2023-07-06T16:00:56 tid=873012 @ process_read_rpc() [unifyfs_request_manager.c:1324] processing mread[0] with 1 requests
2023-07-06T16:00:56 tid=873012 @ submit_read_request() [unifyfs_fops_rpc.c:252] handling read request (1 extents)
2023-07-06T16:00:56 tid=873012 @ pull_margo_bulk_buffer() [../../common/src/unifyfs_rpc_util.c:179] margo_bulk_transfer(buf_offset=0, len=1572864) failed
2023-07-06T16:00:56 tid=873012 @ pull_margo_bulk_buffer() [../../common/src/unifyfs_rpc_util.c:197] failed bulk transfer - transferred 0 of 1572864 bytes
2023-07-06T16:00:56 tid=873012 @ unifyfs_invoke_find_extents_rpc() [unifyfs_p2p_rpc.c:665] failed to get bulk chunk locations
2023-07-06T16:00:56 tid=873012 @ submit_read_request() [unifyfs_fops_rpc.c:279] failed to find extent locations
2023-07-06T16:00:56 tid=873012 @ process_read_rpc() [unifyfs_request_manager.c:1333] unifyfs_fops_read() failed
2023-07-06T16:00:56 tid=873012 @ rm_process_client_requests() [unifyfs_request_manager.c:1690] client rpc request 0 failed ("Mercury/Argobots operation error")
2023-07-06T16:00:56 tid=873012 @ request_manager_thread() [unifyfs_request_manager.c:1768] failed to process client rpc requests
```
The error code returned to the client for the read is 1004. That probably corresponds to one of these:
https://github.com/mercury-hpc/mercury/blob/55b95f72714bb0e4e0deeedf4fd78d116ea9476a/src/mercury_core_types.h#L102-L108
The read error happens during ``PMI_File_read_at_all`` which then leads to a deadlock in ROMIO:
https://github.com/pmodels/mpich/issues/6585
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the four-rank, two-node case using test/largefile/high_dim_var.c and trace the read through unifyfs_request_manager.c, unifyfs_fops_rpc.c, unifyfs_p2p_rpc.c, and common/src/unifyfs_rpc_util.c. Start with the failed bulk transfer and error 1004, then verify that PMI_File_read_at_all completes without the ROMIO deadlock described in MPICH issue 6585.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- distributed-systems, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100