JuliaParallel / JuliaParallel/MPI.jl

Cray MPI autodetection fails on AMD MI300A node (missing `mpi_gtl_hsa` preload)

Open
#919 1 comment 0 reactions 0 assignees View on GitHub
bug Cray MPI
Dominant language
Julia
Stars
439
Forks
127
Avg merge
19h 6m
Merged PRs (30d)
17

Description

On a node with 4× AMD MI300A GPUs, the automatic detection of the setup via
`MPIPreferences.use_system_binary(vendor="cray")` fails.

```julia
julia> MPIPreferences.use_system_binary(vendor="cray")
ERROR: ArgumentError: Collection has multiple elements, must contain exactly 1 element
Stacktrace:
[1] _only
@ ./iterators.jl:1554 [inlined]
[2] only
@ ./iterators.jl:1545 [inlined]
[3] only_or_nothing
@ ~/.julia/packages/MPIPreferences/PLH7x/src/parse_cray_cc.jl:42 [inlined]
[4] cray_gtl
@ ~/.julia/packages/MPIPreferences/PLH7x/src/parse_cray_cc.jl:56 [inlined]
[5] analyze_cray_cc()
@ MPIPreferences.CrayParser ~/.julia/packages/MPIPreferences/PLH7x/src/parse_cray_cc.jl:84
[6] use_system_binary(...)
@ MPIPreferences ~/.julia/packages/MPIPreferences/PLH7x/src/MPIPreferences.jl:180
[7] top-level scope
@ REPL[2]:1
```

Manually editing `LocalPreferences.toml` to include:
```
[MPIPreferences]
_format = "1.1"
abi = "MPICH"
binary = "system"
libmpi = "libmpi_cray"
preloads = ["libmpi_gtl_hsa.so"]
cclibs = []
```
fixes the issue - MPI.jl then runs correctly across GPUs.

The node uses
```
cray-mpich/8.1.30
rocm/6.2.2
```

`$CRAY_MPICH_ROOTDIR/gtl/lib` contains:
```
libmpi_gtl_cuda.a libmpi_gtl_cuda.so libmpi_gtl_cuda.so.0 libmpi_gtl_cuda.so.0.1.0 libmpi_gtl_hsa.a libmpi_gtl_hsa.so libmpi_gtl_hsa.so.0 libmpi_gtl_hsa.so.0.1.0 libmpi_gtl_ze.a libmpi_gtl_ze.so libmpi_gtl_ze.so.0 libmpi_gtl_ze.so.0.1.0 pkgconfig
```

cc @vchuravy

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.