GEOS-DEV / GEOS-DEV/GEOS

Unexpected crash when running simple flow problems with cuda enabled.

Open
#3,846 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

type: bug type: new
Dominant language
C++
Stars
287
Forks
109
Avg merge
4d 41m
Merged PRs (30d)
5

Description

Describe the bug
Runtime crash when running simple cases with cuda compilation.

To Reproduce

  1. Compile with cuda and try to run:
bin/geosx -i ../inputFiles/singlePhaseFlow/sourceFlux_1d.xml -o output 
bin/geosx -i ../inputFiles/singlePhaseFlow/incompressible_1d.xml -o output 

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots

bin/geosx -i ../inputFiles/singlePhaseFlow/sourceFlux_1d.xml -o output 
Num ranks: 1
GEOS version: 1.1.0
  - c++ compiler: gcc 11.4.0
  - CUDA compiler version: 12.8
  - MPI version: Open MPI v4.1.2, package: Debian OpenMPI, ident: 4.1.2, repo rev: v4.1.2, Nov 24, 2021
  - HDF5 version: 1.12.1
  - Conduit version: 0.9.2
  - VTK version: 9.4.2
  - RAJA version: 2025.3.0
  - umpire version: 2025.3.0
  - chai version: 2025.3.0
  - adiak version: ..
  - caliper version: ..
  - METIS version: 5.1.0
  - PARAMETIS version: 4.0.0
  - superlu_dist version: 6.3.0
  - suitesparse version: 5.7.9
  - hypre version: v2.33.0-20-g907a2d07b (HEAD)
  - trilinos version: 16.1.0
  - Python3 version: 3.10.12
  - CUDAToolkit version: 12.8.61
Started at 2025-10-03 22:39:25.189185325
Opened XML file: /workspace/GEOS/inputFiles/singlePhaseFlow/sourceFlux_1d.xml
Solvers: adding SinglePhaseFVM SinglePhaseFlow
Mesh: adding InternalMesh mesh1
Geometry: adding Box source
Geometry: adding Box sink
Events: adding PeriodicEvent outputs
Events: adding PeriodicEvent solverApplications
Events: adding PeriodicEvent restarts
FiniteVolume: adding TwoPointFluxApproximation singlePhaseTPFA
FieldSpecifications: adding FieldSpecification Porosity
FieldSpecifications: adding FieldSpecification initialPressure
FieldSpecifications: adding SourceFlux sourceTerm
FieldSpecifications: adding FieldSpecification sinkTerm
Outputs: adding Silo siloOutput
Outputs: adding Restart restartOutput
Constitutive: adding CompressibleSinglePhaseFluid water
Constitutive: adding CompressibleSolidConstantPermeability rock
Constitutive: adding NullModel nullSolid
Constitutive: adding PressurePorosity rockPorosity
Constitutive: adding ConstantPermeability rockPerm
ElementRegions: adding CellElementRegion Region1

####################################################################################################
##                                        Mesh generation                                         ##
####################################################################################################

mesh1: total number of nodes = 44
mesh1: total number of elems = 10
MPI Partitioning information:

--------------------------------------------------------------------------------------------------------------------
|                                           Mesh partitioning over ranks                                           |
|------------------------------------------------------------------------------------------------------------------|
|      Ranks       |         Nodes         |         Edges         |         Faces         |         Elems         |
|------------------|-----------------------|-----------------------|-----------------------|-----------------------|
|                  | Local | Ghost | Total | Local | Ghost | Total | Local | Ghost | Total | Local | Ghost | Total |
|------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
|                0 |    44 |     0 |    44 |    84 |     0 |    84 |    51 |     0 |    51 |    10 |     0 |    10 |
|------------------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|-------|
|              sum |    44 |     0 |    44 |    84 |     0 |    84 |    51 |     0 |    51 |    10 |     0 |    10 |
|              min |    44 |     0 |    44 |    84 |     0 |    84 |    51 |     0 |    51 |    10 |     0 |    10 |
|              max |    44 |     0 |    44 |    84 |     0 |    84 |    51 |     0 |    51 |    10 |     0 |    10 |
|------------------|-----------------------|-----------------------|-----------------------|-----------------------|
| min(local/total) |                     1 |                     1 |                     1 |                     1 |
| max(local/total) |                     1 |                     1 |                     1 |                     1 |
--------------------------------------------------------------------------------------------------------------------


##                                     End of Mesh generation                                     ##
####################################################################################################


####################################################################################################
##                                       Numerical Methods                                        ##
####################################################################################################

meshBodyName/meshLevelName/regionName/subRegionName = mesh1/Level0/Region1/block1, 1 quadrature point
  ConstitutiveModels/water allocated
  ConstitutiveModels/rock allocated
  ConstitutiveModels/nullSolid allocated
  ConstitutiveModels/rockPorosity allocated
  ConstitutiveModels/rockPerm allocated

##                                    End of Numerical Methods                                    ##
####################################################################################################


####################################################################################################
##                                         Import fields                                          ##
####################################################################################################

mesh1: importing field data from mesh dataset
  mesh level = Level0
  volumic fields on Region1/block1
  mesh level = singlePhaseTPFA
  volumic fields on Region1/block1

##                                      End of Import fields                                      ##
####################################################################################################

Time: 0.00e+00 s, dt: 1000 s, Cycle: 0

####################################################################################################
##                                            TIMESTEP                                            ##
####################################################################################################
##  - Time        : 00h00m00s out of 05h33m20s (0% completed)                                     ##
##                  0 s / 20000 s                                                                 ##
##  - Delta Time  : 00h16m40s (1000 s)                                                            ##
##  - Cycle       : 0                                                                             ##


----------------------------------------------------------
|      The summary of declared fields and coupling       |
|--------------------------------------------------------|
|     |          name          |  comp  |  N global DOF  |
|-----|------------------------|--------|----------------|
|  0  |  singlePhaseVariables  |     1  |            10  |
----------------------------------------------------------

Received signal 11: Segmentation fault

** StackTrace of 16 frames **
Frame 0: /lib/x86_64-linux-gnu/libc.so.6 
Frame 1: hypre_CSRMatrixFnorm 
Frame 2: hypre_CSRMatrixTaggedFnorm 
Frame 3: hypre_ParCSRMatrixComputeScalingTagged 
Frame 4: geos::HypreMatrix::computeScalingVector(geos::HypreVector&) const 
Frame 5: geos::PhysicsSolverBase::solveLinearSystem(geos::DofManager const&, geos::HypreMatrix&, geos::HypreVector&, geos::HypreVector&) 
Frame 6: geos::PhysicsSolverBase::solveNonlinearSystem(double const&, double const&, int, geos::DomainPartition&) 
Frame 7: geos::PhysicsSolverBase::nonlinearImplicitStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 8: geos::PhysicsSolverBase::solverStep(double const&, double const&, int, geos::DomainPartition&) 
Frame 9: geos::PhysicsSolverBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 10: geos::EventBase::execute(double, double, int, int, double, geos::DomainPartition&) 
Frame 11: geos::EventManager::run(geos::DomainPartition&) 
Frame 12: geos::GeosxState::run() 
Frame 13: main 
Frame 14: /lib/x86_64-linux-gnu/libc.so.6 
Frame 15: __libc_start_main 
Frame 16: _start 
=====

Platform (please complete the following information):

  • Machine: H200
  • Compiler: gcc 11.4]
  • GEOS Version: latest develop
  • Cuda 12.8

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the crash with bin/geosx using inputFiles/singlePhaseFlow/sourceFlux_1d.xml and incompressible_1d.xml in a CUDA build. Start at geos::HypreMatrix::computeScalingVector and PhysicsSolverBase::solveLinearSystem, then compare the hypre CSR stack frames; done means both simple flows run without the segmentation fault.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.