`libfabric=1.17.0-3` on Debian causes MPI tests to fail with `MPI_ERR_OTHER`
- Dominant language
- Python
- Stars
- 34
- Forks
- 25
- Avg merge
- 3h 37m
- Merged PRs (30d)
- 2
Description
Sample CI failure:
https://gitlab.tiker.net/inducer/meshmode/-/jobs/533461
Similar failure in grudge:
https://gitlab.tiker.net/inducer/grudge/-/jobs/533485
Sample traceback:
```
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/.env/lib/python3.11/site-packages/mpi4py/run.py", line 208, in
main()
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/.env/lib/python3.11/site-packages/mpi4py/run.py", line 198, in main
run_command_line(args)
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/.env/lib/python3.11/site-packages/mpi4py/run.py", line 47, in run_command_line
run_path(sys.argv[0], run_name='__main__')
File "", line 291, in run_path
File "", line 98, in _run_module_code
File "", line 88, in _run_code
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/test/test_partition.py", line 609, in
_test_mpi_boundary_swap(dim, order, num_groups)
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/test/test_partition.py", line 426, in _test_mpi_boundary_swap
conns = bdry_setup_helper.complete_some()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/meshmode/distributed.py", line 332, in complete_some
data = [self._internal_mpi_comm.recv(status=status)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "mpi4py/MPI/Comm.pyx", line 1438, in mpi4py.MPI.Comm.recv
File "mpi4py/MPI/msgpickle.pxi", line 341, in mpi4py.MPI.PyMPI_recv
File "mpi4py/MPI/msgpickle.pxi", line 303, in mpi4py.MPI.PyMPI_recv_match
mpi4py.MPI.Exception: MPI_ERR_OTHER: known error not in list
```
Downgrading to libfabric (see [here](https://github.com/illinois-scicomp/machine-shop-maintenance/commit/f42ef5f708c9de7dff28b022407ffe2da4b32aba)) appears to resolve this.
[This](https://github.com/mpi4py/mpi4py/blob/a7610e53b78841e1a607005d5b3ace7b9315f45b/src/mpi4py/MPI/msgpickle.pxi#L286-L306) is the code in mpi4py that ultimately fails, it's a matched receive (`mrecv`).
@majosm Got any ideas? (Pinging you since the two of us last touched this code.)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.