[Bug]: segfault on readout app at shutdown
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 0
- Forks
- 1
- Avg merge
- 5h 38m
- Merged PRs (30d)
- 4
Description
Description
Description
A segfault occurs during application shutdown on np02-srv-003 running several dune daq versions, including v.5.6.0 as reported in op meetings (20, 27th of July).
The crash happens after the application has already logged a clean exit, so data integrity is not affected, but the signal 11 is noisy and may mask real issues.
Stack trace
2026-Aug-11 17:48:26,235 LOG [void dunedaq::opmonlib::OpMonManager::run(std::stop_token) at /tmp/root/spack-stage/spack-stage-opmonlib-v2.1.6-6kgxctp7miol4lqfk7k3xjkmzw4o7oun/spack-src/src/OpMonManager.cpp:75] Exiting the monitoring thread
2026-Aug-11 17:48:26,235 LOG [int main(int, char**) at /tmp/root/spack-stage/spack-stage-appfwk-v6.1.1-gbgr7tsjxulzolypyd6cbgrj4iqtnjr3/spack-src/apps/daq_application.cxx:79] Application main-np02-ops-pns.runp02srv003eth1 exiting.
2026-Aug-11 17:48:27,657 ERROR [static void ers::ErrorHandler::SignalHandler::action(int, siginfo_t*, void*) at /tmp/root/spack-stage/spack-stage-ers-v1.5.5-hkrj5g33t5yyjv7pyn74pdtcad4cex22/spack-src/src/ErrorHandler.cpp:90] Got signal 11 Segmentation fault (invalid memory reference)
Parameters = 'name=Segmentation fault (invalid memory reference)' 'signum=11'
Qualifiers = 'unknown'
host = np02-srv-003
user = np04daq (106200)
process id = 3287102
thread id = 3287102
process wd = /nfs/sw/dunedaq/dunedaq-fddaq-v5.6.2-dev-ops-pns
stack trace of the crashing thread:
#0 /lib64/libc.so.6(free+0x1e) [0x7fcd652af91e]
#1 /lib64/libc.so.6(+0x54db0) [0x7fcd65254db0]
#2 /lib64/libc.so.6(free+0x1e) [0x7fcd652af91e]
#3 /cvmfs/dunedaq.opensciencegrid.org/spack/releases/fddaq-v5.6.2-a9-1/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/dpdklibs-v2.3.8-bzsbnyoww5v76u2sgerxsbebw77nzbbz/lib64/libdpdklibs.so(dunedaq::dpdklibs::IfaceWrapper::~IfaceWrapper()+0x6ed) [0x7fcd558a35fd]
#4 daq_application(std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()+0xa) [0x4423da]
#5 /cvmfs/dunedaq.opensciencegrid.org/spack/releases/fddaq-v5.6.2-a9-1/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/dpdklibs-v2.3.8-bzsbnyoww5v76u2sgerxsbebw77nzbbz/lib64/libdpdklibs_DPDKReaderModule_duneDAQModule.so(dunedaq::dpdklibs::DPDKReaderModule::~DPDKReaderModule()+0x85) [0x7fcd55c3d465]
#6 /cvmfs/dunedaq.opensciencegrid.org/spack/releases/fddaq-v5.6.2-a9-1/spack-0.22.0/opt/spack/linux-almalinux9-x86_64/gcc-14.3.0/dpdklibs-v2.3.8-bzsbnyoww5v76u2sgerxsbebw77nzbbz/lib64/libdpdklibs_DPDKReaderModule_duneDAQModule.so(std::_Sp_counted_ptr<dunedaq::dpdklibs::DPDKReaderModule*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x25) [0x7fcd55c7ece5]
#7 daq_application(std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()+0xa) [0x4423da]
#8 daq_application() [0x445d52]
#9 daq_application(dunedaq::appfwk::Application::~Application()+0xee) [0x4586fe]
#10 daq_application() [0x434a42]
#11 /lib64/libc.so.6(+0x3feb0) [0x7fcd6523feb0]
#12 /lib64/libc.so.6(__libc_start_main+0x80) [0x7fcd6523ff60]
#13 daq_application() [0x435345]
~
"./log_np04daq_main-np02-ops-pns_runp02srv003eth1_2026-08-11-10-54-27.txt" [readonly] 1578L, 684606B
Steps to reproduce
the error will appear every now and then at shutdown.
Version and environment
dunedaq-fddaq-v5.6.0-dev
Impact
Small/Isolated
Anything else?
One can see a log here (also a printout below):
np02-srv-003:
/log//log_np04daq_main-np02-ops-pns_runp02srv003eth1_2026-08-11-10-54-27.txt
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the DPDKReaderModule and IfaceWrapper destructors named in the stack trace, then inspect the shutdown path through apps/daq_application.cxx and OpMonManager.cpp. Reproduce shutdown on the reported DUNE DAQ versions and use the provided signal-11 trace to isolate the invalid free. Done means the application exits without the intermittent segfault while preserving the clean shutdown behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100