DUNE-DAQ / DUNE-DAQ/dpdklibs

[Bug]: segfault on readout app at shutdown

Open
#197 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C++
Stars
0
Forks
1
Avg merge
5h 38m
Merged PRs (30d)
4

Description

Description
Description

A segfault occurs during application shutdown on np02-srv-003 running several dune daq versions, including v.5.6.0 as reported in op meetings (20, 27th of July).

The crash happens after the application has already logged a clean exit, so data integrity is not affected, but the signal 11 is noisy and may mask real issues.

Stack trace
2026-Aug-11 17:48:26,235 LOG [void dunedaq::opmonlib::OpMonManager::run(std::stop_token) at /tmp/root/spack-stage/spack-stage-opmonlib-v2.1.6-6kgxctp7miol4lqfk7k3xjkmzw4o7oun/spack-src/src/OpMonManager.cpp:75] Exiting the monitoring thread
2026-Aug-11 17:48:26,235 LOG [int main(int, char**) at /tmp/root/spack-stage/spack-stage-appfwk-v6.1.1-gbgr7tsjxulzolypyd6cbgrj4iqtnjr3/spack-src/apps/daq_application.cxx:79] Application main-np02-ops-pns.runp02srv003eth1 exiting.
2026-Aug-11 17:48:27,657 ERROR [static void ers::ErrorHandler::SignalHandler::action(int, siginfo_t*, void*) at /tmp/root/spack-stage/spack-stage-ers-v1.5.5-hkrj5g33t5yyjv7pyn74pdtcad4cex22/spack-src/src/ErrorHandler.cpp:90] Got signal 11 Segmentation fault (invalid memory reference)
        Parameters = 'name=Segmentation fault (invalid memory reference)' 'signum=11'
        Qualifiers = 'unknown'
        host = np02-srv-003
        user = np04daq (106200)
        process id = 3287102
        thread id = 3287102
        process wd = /nfs/sw/dunedaq/dunedaq-fddaq-v5.6.2-dev-ops-pns
        stack trace of the crashing thread:
          #0  /lib64/libc.so.6(free+0x1e) [0x7fcd652af91e]
          #1  /lib64/libc.so.6(+0x54db0) [0x7fcd65254db0]
          #2  /lib64/libc.so.6(free+0x1e) [0x7fcd652af91e]
          #3  /cvmfs/dunedaq.opensciencegrid.org/spack/releases/fddaq-v5.6.2-a9-1/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/dpdklibs-v2.3.8-bzsbnyoww5v76u2sgerxsbebw77nzbbz/lib64/libdpdklibs.so(dunedaq::dpdklibs::IfaceWrapper::~IfaceWrapper()+0x6ed) [0x7fcd558a35fd]
          #4  daq_application(std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()+0xa) [0x4423da]
          #5  /cvmfs/dunedaq.opensciencegrid.org/spack/releases/fddaq-v5.6.2-a9-1/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/dpdklibs-v2.3.8-bzsbnyoww5v76u2sgerxsbebw77nzbbz/lib64/libdpdklibs_DPDKReaderModule_duneDAQModule.so(dunedaq::dpdklibs::DPDKReaderModule::~DPDKReaderModule()+0x85) [0x7fcd55c3d465]
          #6  /cvmfs/dunedaq.opensciencegrid.org/spack/releases/fddaq-v5.6.2-a9-1/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/dpdklibs-v2.3.8-bzsbnyoww5v76u2sgerxsbebw77nzbbz/lib64/libdpdklibs_DPDKReaderModule_duneDAQModule.so(std::_Sp_counted_ptr<dunedaq::dpdklibs::DPDKReaderModule*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x25) [0x7fcd55c7ece5]
          #7  daq_application(std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()+0xa) [0x4423da]
          #8  daq_application() [0x445d52]
          #9  daq_application(dunedaq::appfwk::Application::~Application()+0xee) [0x4586fe]
          #10 daq_application() [0x434a42]
          #11 /lib64/libc.so.6(+0x3feb0) [0x7fcd6523feb0]
          #12 /lib64/libc.so.6(__libc_start_main+0x80) [0x7fcd6523ff60]
          #13 daq_application() [0x435345]
~
"./log_np04daq_main-np02-ops-pns_runp02srv003eth1_2026-08-11-10-54-27.txt" [readonly] 1578L, 684606B
Steps to reproduce

the error will appear every now and then at shutdown.

Version and environment

dunedaq-fddaq-v5.6.0-dev

Impact

Small/Isolated

Anything else?

One can see a log here (also a printout below):
np02-srv-003:
/log//log_np04daq_main-np02-ops-pns_runp02srv003eth1_2026-08-11-10-54-27.txt

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the DPDKReaderModule and IfaceWrapper destructors named in the stack trace, then inspect the shutdown path through apps/daq_application.cxx and OpMonManager.cpp. Reproduce shutdown on the reported DUNE DAQ versions and use the provided signal-11 trace to isolate the invalid free. Done means the application exits without the intermittent segfault while preserving the clean shutdown behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.