llnl / llnl/smith

umpire runtime error "Cannot remove 0x1ba37f50" when running `test_dfem_explicit_dynamics` with GPU exec space

Open
#1,546 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug gpu
Dominant language
C++
Stars
245
Forks
36
Avg merge
8d 23h
Merged PRs (30d)
4

Description

if

```cpp
smith::ApplicationManager applicationManager(argc, argv, MPI_COMM_WORLD, true, smith::ExecutionSpace::GPU);
```

https://github.com/llnl/smith/blob/develop/src/smith/physics/tests/test_dfem_explicit_dynamics.cpp#L275

then

```
[meemee@rzadams1017:build-rzadams-toss_4_x86_64_ib_cray-llvm-amdgpu@6.4.2_hip-release]$ f1 ./tests/test_dfem_explicit_dynamics
flux run -N1 -x ./tests/test_dfem_explicit_dynamics
flux-job: f3WagnVzSyBd started 00:00:04
Logger activated: 'smith_serial_logger'
********************************************************************************
Smith Version: v0.0.1-95d795dcb
Build Type: Release
User Name: meemee
Host Name: rzadams1050
MPI Rank Count: 1
********************************************************************************

[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ExplicitDynamicsFixture
[ RUN ] ExplicitDynamicsFixture.RunDfemExplicitDynamicsSim
Step 0, time = 0
Step 1, time = 0.001
Step 2, time = 0.002
[ OK ] ExplicitDynamicsFixture.RunDfemExplicitDynamicsSim (266 ms)
[----------] 1 test from ExplicitDynamicsFixture (266 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test suite ran. (266 ms total)
[ PASSED ] 1 test.
terminate called after throwing an instance of 'umpire::runtime_error'
what(): ! Umpire runtime_error [/usr/WS2/smithdev/libs/smith/toss_4_x86_64_ib_cray/2026_02_13_17_24_00/build_stage/meemee/spack-stage-umpire-2025.09.0-ifzxz2gfr6lwe2noancq2srjpbwxvhot/spack-src/src/umpire/util/AllocationMap.cpp:255]: Cannot remove 0x1ba37f50
Backtrace: 15 frames
0 0xddc975 No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN6umpire4util10backtracerINS0_12trace_alwaysEE13get_backtraceERNS0_9backtraceE+0x25) [0xddc975]
1 0xddc5dc No dladdr: ./tests/test_dfem_explicit_dynamics(_ZNK6umpire13runtime_error7messageB5cxx11Ev+0x2c) [0xddc5dc]
2 0xddb09b No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN6umpire13runtime_errorC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_i+0x13b) [0xddb09b]
3 0xe3b572 No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN6umpire4util13AllocationMap6removeEPv+0x4b2) [0xe3b572]
4 0xdd27b1 No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN6umpire15ResourceManager20deregisterAllocationEPv+0x251) [0xdd27b1]
5 0xe2193a No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN6umpire8strategy6mixins9Inspector20deregisterAllocationEPvPNS0_18AllocationStrategyE+0x2a) [0xe2193a]
6 0xe0652e No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN6umpire9Allocator13do_deallocateEPv+0x28e) [0xe0652e]
7 0x143ccb3 No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN4mfem13MemoryManager7DestroyEv+0x93) [0x143ccb3]
8 0x1460303 No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN4mfem6DeviceD1Ev+0x33) [0x1460303]
9 0xd309d0 No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN5smith11accelerator15terminateDeviceEv+0x20) [0xd309d0]
10 0xd2fbe6 No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN5smith9finalizerEv+0x56) [0xd2fbe6]
11 0xd2fda6 No dladdr: ./tests/test_dfem_explicit_dynamics(_ZN5smith18ApplicationManagerD1Ev+0x6) [0xd2fda6]
12 0xc26271 No dladdr: ./tests/test_dfem_explicit_dynamics(main+0x51) [0xc26271]
13 0x1554e1dcb865 No dladdr: /lib64/libc.so.6(__libc_start_main+0xe5) [0x1554e1dcb865]
14 0xc24a1e No dladdr: ./tests/test_dfem_explicit_dynamics(_start+0x2e) [0xc24a1e]

[SIGNAL]: Received signal 6 (Aborted), exiting
terminate called recursively
```

its possible the memory its trying to remove has already been free'd but not sure.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/smith/physics/tests/test_dfem_explicit_dynamics.cpp at the ApplicationManager GPU setup and reproduce with tests/test_dfem_explicit_dynamics. Trace the teardown path shown in the backtrace from ApplicationManager destruction through finalizer, terminateDevice, and mfem::Device destruction. Done means the test exits without the Umpire Cannot remove error while retaining the passing simulation test.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.