llnl / llnl/GOTCHA

GOTCHA segfaults in GEOS on MI300C

Open
#167 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
88
Forks
31
PR merge metrics
No merged PRs in 30d

Description

Hi! I'm investigating a segfault in GOTCHA during Caliper initialization. Here's the original discussion thread: https://github.com/LLNL/Caliper/discussions/678

Here's the gist:
- The code segfaults when Caliper initializes its MPI wrappers. The segfault seems to occur at some point during GOTCHA initialization while wrapping the dlsym/dlopen functions. It doesn't get to the point where it says "gotcha wrap completed successfully" for the dlsym/dlopen gotcha_wrap call.
- It happens in a larger MPI program but does not segfault with a small test app.
- The system is a 384-core machine on Azure, so I don't know if we can get direct access to it

The log shows many errors like
```
ERROR [86136/86136][gotcha.c:324] - GOTCHA attempted to mark both GOT and PLT GOT tables as writable and was unable to do so, calls to wrapped functions may likely fail.
[86136/86136][gotcha.c:316] - Setting library /home/hpcuser/miket/software/x86_64/RHEL9/GEOS/1.1.0--victor-caliper/install-CPU-MI300C-Hypre-GCC-mpi-OMP-relwithdebinfo/lib/libmainInterface.so GOT and PLT table from 0x7ffff7fba000 to +16384 to writeable
ERROR [86136/86136][gotcha.c:324] - GOTCHA attempted to mark both GOT and PLT GOT tables as writable and was unable to do so, calls to wrapped functions may likely fail.
[86136/86136][gotcha.c:316] - Setting library /home/hpcuser/miket/software/x86_64/RHEL9/GEOS/1.1.0--victor-caliper/install-CPU-MI300C-Hypre-GCC-mpi-OMP-relwithdebinfo/lib/libphysicsSolvers.so GOT and PLT table from 0x7ffff7e6c000 to +8192 to writeable
```
However they also appear in the small test example which didn't segfault.

Here's a stack trace:
```
Received signal 11: Segmentation fault

** StackTrace of 16 frames **
Frame 0: /lib64/libc.so.6
Frame 1: /home/hpcuser/miket/software/x86_64/RHEL9/GEOSTPL/1.1.0--update-hypre/install-CPU-MI300C-Hypre-GCC-mpi-OMP-relwithdebinfo/caliper/lib64/libcaliper.so.2
Frame 2: update_all_library_gots
Frame 3: gotcha_wrap
Frame 4: gotcha_wrap
Frame 5: cali::mpiwrap_init(cali::Caliper*, cali::Channel*, cali::ConfigSet&)
Frame 6: /home/hpcuser/miket/software/x86_64/RHEL9/GEOSTPL/1.1.0--update-hypre/install-CPU-MI300C-Hypre-GCC-mpi-OMP-relwithdebinfo/caliper/lib64/libcaliper.so.2
Frame 7: cali::services::register_configured_services(cali::Caliper*, cali::Channel*)
Frame 8: cali::Caliper::create_channel(char const*, cali::RuntimeConfig const&)
Frame 9: cali::ChannelController::create()
Frame 10: cali::ChannelController::start()
Frame 11: cali::ConfigManager::start()
Frame 12: geos::GeosxState::GeosxState(std::unique_ptr >&&)
Frame 13: main
Frame 14: /lib64/libc.so.6
Frame 15: __libc_start_main
Frame 16: _start
=====
```

There are full level 3 debug logs in the discussion thread linked above. Doesn't really point me to anything suspicious though other than those error messages.

Any ideas on what might be going on or any tips on how to debug this?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with gotcha.c around lines 316-324 and trace update_all_library_gots and gotcha_wrap during the Caliper mpiwrap_init path shown in the stack trace. Compare the small test application with the larger MPI program using the level 3 logs from discussion 678. Done means identifying the initialization failure and validating a fix or a reproducible diagnosis on the affected configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.