DUNE-DAQ / DUNE-DAQ/daqsystemtest
Investigate intermittent regression test failures
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 9
Description
Here is a sampling of intermittent problems that have been seen recently.
-
The
tpstream_writing_testin the 16-Aug-2025 overnight running of the regression tests had a "failed to insert into LB" message in one of its log files.- Here is a copy of the message:
There were 1 failures to insert data into the latency buffer out of 4887 attempts in the latest monitoring interval. - Here is a link to the GitHub CI log: link
- Here is a copy of the message:
-
The
example_system_testin the 09-Aug-2025 overnight running of the regression tests had a "tp_input_2000: Unable to push within timeout period" message in one of its log files (link).- I've seen this problem on other regression tests.
- When I get a chance, I'd like to propose some tests with mutex locking in our "queue" code...
- This was also seen in the
3ru_1dftest on 12-Sep-2025 (link) - And in the DAPHNE_TPG part of the
readout_type_scan_teston 15-Sep-2025 (link) - And here
- And in the 3ru_1df test on 11-Oct-2025 (link)
-
I've seen the number of TriggerRecords produced in the
example_system_testoccasionally fluctuate slightly too low.- That happened in the 08-Aug-2025 overnight running of the regression tests. (link)
- I saw it in a running of all regression tests using
dbt-build --integteston the fddaq-v5.4.0-rc1 candidate release
-
The DAPHNE_TPG part of the
readout_type_scan_testcomplained about an empty TP fragment in the 04-Sep-2025 overnight running of the regression tests.- Here is a copy of the error message:
Trigger Primitive fragment for SrcID Trigger_0x00000064 in record (5, 0) has size 72 (outside range [96, 4392])
- Here is a copy of the error message:
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the linked GitHub CI logs for tpstream_writing_test, example_system_test, 3ru_1df, and readout_type_scan_test, then compare their reported failures across runs. Done means identifying reproducible causes or contributing conditions for the intermittent insert, timeout, TriggerRecord-count, and empty-fragment failures, with clear follow-up scope.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100