Simulation with two internal mesh bodies crashes in parallel
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 287
- Forks
- 109
- Avg merge
- 4d 41m
- Merged PRs (30d)
- 5
Description
Describe the bug
Crash when running an xml with two internal mesh bodies in parallel (any number of ranks >1).
To Reproduce
Steps to reproduce the behavior:
- Check out
feature/andrembcosta/multiBodyVTKOutputbranch if https://github.com/GEOSX/GEOSX/pull/2027 is not yet merged, ordevelop - Run
mpirun -n 2 geosx -i inputFiles/multipleMeshBodies/testMultipleBodies.xml -x 2 - Get a crash on rank 1:
Received signal 8: Floating point exception
Floating point exception:
** StackTrace of 9 frames **
Frame 0: /lib/x86_64-linux-gnu/libc.so.6
Frame 1: void geosx::InternalMeshGenerator::getNodePosition<LvArray::ArraySlice<double, 1, 0, int> >(int const (&) [3], int, LvArray::ArraySlice<double, 1, 0, int>&&)
Frame 2: geosx::InternalMeshGenerator::generateMesh(geosx::DomainPartition&)
Frame 3: geosx::MeshManager::generateMeshes(geosx::DomainPartition&)
Frame 4: geosx::ProblemManager::generateMesh()
Frame 5: geosx::ProblemManager::problemSetup()
Frame 6: geosx::GeosxState::initializeDataRepository()
Frame 7: main
Frame 8: __libc_start_main
Frame 9: _start
=====
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 1 in communicator MPI_COMM_WORLD
with errorcode 1.
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
Expected behavior
Should run (it does in serial).
Platform (please complete the following information):
- Machine: desktop, Ubuntu 20.04
- Compiler: gcc 8.4.0
- GEOSX version 0.2.0 (feature/andrembcosta/multiBodyVTKOutput, sha1: 37dc49e6c)
Additional context
I suspect (haven't investigated yet), that this is related to two different instances of InternalMeshGenerator interacting with the same SpatialPartition object. Each one tries to call partition.setSizes with its own bounds. I assume SpatialPartition was not written with multiple mesh bodies in mind and does something invalid. If either of the two meshes is commented out, mesh generation completes without a crash. It's not immediately obvious to me how the domain and each of the bodies should be partitioned in the general case when their physical extents don't match.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce with mpirun -n 2 geosx -i inputFiles/multipleMeshBodies/testMultipleBodies.xml -x 2 and inspect the stack path through InternalMeshGenerator::getNodePosition, generateMesh, and MeshManager::generateMeshes. Investigate how multiple InternalMeshGenerator instances use the shared SpatialPartition and call setSizes. Done means the two internal mesh bodies complete mesh generation in parallel without the rank-1 floating-point crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100