boostorg / boostorg/mpi

mpi::reduce hangs on intel MPI

Open
#124 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
64
Forks
65
PR merge metrics
No merged PRs in 30d

Description

The following program hangs on calling ```boost::mpi::reduce``` when run on an Intel MPI environment.

```
#include
#include
#include
#include
#include
#include

struct sum_vec_vec {
std::vector operator()(const std::vector& a, const std::vector&b) const
{
std::vector res(a.size());
std::transform(a.begin(), a.end(), b.begin(), res.begin(), [](double x, double y) { return x + y; });
return res;
}
};

namespace boost {
namespace mpi {
template <>
struct is_commutative> : mpl::true_ { };
}
}

int main()
{
namespace mpi = boost::mpi;

mpi::environment env;
mpi::communicator world;

std::size_t size = 1000;
std::size_t L = 32;

std::vector> correlations;

// Fill up with some data
for (std::size_t i = 0; i < size; ++i)
{
int l = 0;

std::vector corr(L);
for (auto&x : corr)
x = l++;
correlations.emplace_back(std::move(corr));
}

std::vector> av_correlations(correlations.size());
std::cout << "Ready for mpi::reduce" << std::endl;
boost::mpi::reduce(world, &correlations.front(), correlations.size(), &av_correlations.front(), sum_vec_vec{}, 0);

return 0;
}

```

The specific of the MPI environment are:

```
MPI_Get_library_version: Intel(R) MPI Library 2019 Update 6 for Linux* OS
MPI_VERSION: 3
I_MPI_NUMVERSION: 20190006300
```

Boost version is 1.74.0, and it defines:

```
BOOST_MPI_VERSION: 3
BOOST_MPI_USE_IMPROBE: 1
```

The program above very often hangs when run for example with >6 tasks, and always hangs when, say, Ntasks=192.

The reason apparently lies in the use of the ```MPI_Mprobe``` routines in [point_to_point.cpp](https://github.com/boostorg/mpi/blob/070c3c707d50fee0515e00d2e2c9a5744f10bbba/src/point_to_point.cpp#L76).
On recompiling the library with the flag ```BOOST_MPI_USE_IMPROBE``` disabled in [config.hpp](https://github.com/boostorg/mpi/blob/ace80263884ee2af3cef2ab3644ff4b41d154e49/include/boost/mpi/config.hpp#L137), the program ends without issues. Also, the program runs without problems, on openmpi and enabled ```BOOST_MPI_USE_IMPROBE```.

All in all, I suspect that the bug mentioned [here](https://github.com/boostorg/mpi/issues/63#issuecomment-415905771) is still there.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.