collective operations with large data sizes
- Dominant language
- C++
- Stars
- 64
- Forks
- 65
- PR merge metrics
- No merged PRs in 30d
Description
The library crashes when performing a collective operation like ```gather``` when the size of the objects is very large, but that should be still manageable with supercomputers. The issue is not easy to reproduce, because it requires quite some memory available.
The following programs illustrates this:
```
#include
#include
#include
#include
struct huge {
std::vector data;
huge() : data(2ull << 30ull, 0) { }
template
void serialize(Archive& ar, const unsigned int version)
{
ar & data;
}
};
int main()
{
boost::mpi::environment env;
boost::mpi::communicator world;
huge a{};
std::cout << world.rank() << " huge created " << std::endl;
world.barrier();
if (world.rank() == 0)
{
std::vector all;
boost::mpi::gather(world, a, all, 0);
}
else
{
boost::mpi::gather(world, a, 0);
}
return 0;
}
```
The program create an object of 1G of memory. The struct ```huge``` is defined so to force the library to have a non-primitive MPI type. When run with only 2 tasks, it crashes giving
```
terminate called after throwing an instance of 'std::length_error'
what(): vector::_M_range_insert
```
On a supercomputer JUWELS, which has boost 1.69, the error is:
```
mpi: /gpfs/software/juwels/stages/2019a/software/Boost/1.69.0-gpsmpi-2019a-Python-2.7.16/include/boost/mpi/allocator.hpp:142: T* boost::mpi::allocator::allocate(boost::mpi::allocator::size_type, boost::mpi::allocator::const_pointer) [with T = char; boost::mpi::allocator::pointer = char*; boost::mpi::allocator::size_type = long unsigned int; boost::mpi::allocator::const_pointer = const void*]: Assertion `_check_result == 0' failed.
```
It appears that the programs crashes around [this line](https://github.com/boostorg/mpi/blob/04743c9bfbbb4cb5c1f080728ad6db53b4a0a00a/include/boost/mpi/collectives/gather.hpp#L66) on ```gather.hpp```
oa << in_values[i];
The same crash is found even running with 1 task.
Reducing the size of ```huge``` as
```
struct huge {
std::vector data;
huge() : data(2ull << 29ull, 0) { }
....
```
makes again the program crash, this time it appears that the crash happens at [this line](https://github.com/boostorg/mpi/blob/04743c9bfbbb4cb5c1f080728ad6db53b4a0a00a/include/boost/mpi/collectives/gather.hpp#L81) of ```gather.hpp```
```
packed_iarchive::buffer_type recv_buffer(is_root ? std::accumulate(oasizes.begin(), oasizes.end(), 0) : 0);
```
My impression is that the sizes are sometimes stored as ```int``` when they should be a ```size_t```. For example, in the line above, ```oasizes``` is a ```std::vector``` of ```int```: even if the single-object size fits into a ```int```, the total buffer of gathered objects could exceed 2^31.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.