inducer / inducer/meshmode

`libfabric=1.17.0-3` on Debian causes MPI tests to fail with `MPI_ERR_OTHER`

Open
#370 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
34
Forks
25
Avg merge
3h 37m
Merged PRs (30d)
2

Description

Sample CI failure:
https://gitlab.tiker.net/inducer/meshmode/-/jobs/533461

Similar failure in grudge:
https://gitlab.tiker.net/inducer/grudge/-/jobs/533485

Sample traceback:
```
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/.env/lib/python3.11/site-packages/mpi4py/run.py", line 208, in
main()
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/.env/lib/python3.11/site-packages/mpi4py/run.py", line 198, in main
run_command_line(args)
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/.env/lib/python3.11/site-packages/mpi4py/run.py", line 47, in run_command_line
run_path(sys.argv[0], run_name='__main__')
File "", line 291, in run_path
File "", line 98, in _run_module_code
File "", line 88, in _run_code
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/test/test_partition.py", line 609, in
_test_mpi_boundary_swap(dim, order, num_groups)
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/test/test_partition.py", line 426, in _test_mpi_boundary_swap
conns = bdry_setup_helper.complete_some()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/var/lib/gitlab-runner/builds/VFPjm48d/0/inducer/meshmode/meshmode/distributed.py", line 332, in complete_some
data = [self._internal_mpi_comm.recv(status=status)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "mpi4py/MPI/Comm.pyx", line 1438, in mpi4py.MPI.Comm.recv
File "mpi4py/MPI/msgpickle.pxi", line 341, in mpi4py.MPI.PyMPI_recv
File "mpi4py/MPI/msgpickle.pxi", line 303, in mpi4py.MPI.PyMPI_recv_match
mpi4py.MPI.Exception: MPI_ERR_OTHER: known error not in list
```
Downgrading to libfabric (see [here](https://github.com/illinois-scicomp/machine-shop-maintenance/commit/f42ef5f708c9de7dff28b022407ffe2da4b32aba)) appears to resolve this.

[This](https://github.com/mpi4py/mpi4py/blob/a7610e53b78841e1a607005d5b3ace7b9315f45b/src/mpi4py/MPI/msgpickle.pxi#L286-L306) is the code in mpi4py that ultimately fails, it's a matched receive (`mrecv`).

@majosm Got any ideas? (Pinging you since the two of us last touched this code.)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.