UCX protocol framework triggers SIGFPE
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 287
- Forks
- 109
- Avg merge
- 4d 41m
- Merged PRs (30d)
- 5
Description
We tried to build the whole application incl TPLs with AMD software stack --> AOCC,AOCL,OPENMPI built with AOCC.
The build was successful, however multiple unit tests specifically mpi related failed with SIGFPE.
92 - testMpiWrapper_mpi (Failed)234 - testMixedDimSinglePhaseFlow_mpi (Failed)236 - testMixedDimHydrostaticEquilibrium_mpi (Failed)
UCX version: 1.19.0
Git revision: e463614
Failing function:
ucp_proto_multi_init()
Workaround:
export UCX_PROTO_ENABLE=n
Result: Tests passes reliably
Environment: AOCC 5.1.0
We would like guidance on whether this is a known UCX 1.19 issue and whether disabling the protocol framework is expected to impact performance for GEOS runs.
Please see the full stack trace attached.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the attached ucx_sigfpe.log and the failing ucp_proto_multi_init() path. Reproduce testMpiWrapper_mpi, testMixedDimSinglePhaseFlow_mpi, and testMixedDimHydrostaticEquilibrium_mpi with UCX 1.19.0, then compare runs with UCX_PROTO_ENABLE enabled and disabled. Done should identify whether the failure is attributable to UCX and document the performance impact of the workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems, hpc, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100