boostorg / boostorg/mpi

collective operations with large data sizes

Open
#122 10 comments 0 reactions 1 assignee Claimed by @aminiussi View on GitHub
Dominant language
C++
Stars
64
Forks
65
PR merge metrics
No merged PRs in 30d

Description

The library crashes when performing a collective operation like ```gather``` when the size of the objects is very large, but that should be still manageable with supercomputers. The issue is not easy to reproduce, because it requires quite some memory available.

The following programs illustrates this:

```
#include
#include
#include
#include

struct huge {
std::vector data;
huge() : data(2ull << 30ull, 0) { }

template
void serialize(Archive& ar, const unsigned int version)
{
ar & data;
}
};

int main()
{
boost::mpi::environment env;
boost::mpi::communicator world;

huge a{};

std::cout << world.rank() << " huge created " << std::endl;
world.barrier();

if (world.rank() == 0)
{
std::vector all;
boost::mpi::gather(world, a, all, 0);
}
else
{
boost::mpi::gather(world, a, 0);
}

return 0;
}
```

The program create an object of 1G of memory. The struct ```huge``` is defined so to force the library to have a non-primitive MPI type. When run with only 2 tasks, it crashes giving

```
terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_range_insert
```

On a supercomputer JUWELS, which has boost 1.69, the error is:

```
mpi: /gpfs/software/juwels/stages/2019a/software/Boost/1.69.0-gpsmpi-2019a-Python-2.7.16/include/boost/mpi/allocator.hpp:142: T* boost::mpi::allocator::allocate(boost::mpi::allocator::size_type, boost::mpi::allocator::const_pointer) [with T = char; boost::mpi::allocator::pointer = char*; boost::mpi::allocator::size_type = long unsigned int; boost::mpi::allocator::const_pointer = const void*]: Assertion `_check_result == 0' failed.
```
It appears that the programs crashes around [this line](https://github.com/boostorg/mpi/blob/04743c9bfbbb4cb5c1f080728ad6db53b4a0a00a/include/boost/mpi/collectives/gather.hpp#L66) on ```gather.hpp```

oa << in_values[i];

The same crash is found even running with 1 task.

Reducing the size of ```huge``` as

```
struct huge {
std::vector data;
huge() : data(2ull << 29ull, 0) { }

....
```

makes again the program crash, this time it appears that the crash happens at [this line](https://github.com/boostorg/mpi/blob/04743c9bfbbb4cb5c1f080728ad6db53b4a0a00a/include/boost/mpi/collectives/gather.hpp#L81) of ```gather.hpp```

```
packed_iarchive::buffer_type recv_buffer(is_root ? std::accumulate(oasizes.begin(), oasizes.end(), 0) : 0);
```

My impression is that the sizes are sometimes stored as ```int``` when they should be a ```size_t```. For example, in the line above, ```oasizes``` is a ```std::vector``` of ```int```: even if the single-object size fits into a ```int```, the total buffer of gathered objects could exceed 2^31.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.