DUNE-DAQ / DUNE-DAQ/daqsystemtest

Investigate intermittent regression test failures

Open
#238 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1
Forks
0
Avg merge
3d 1h
Merged PRs (30d)
9

Description

Here is a sampling of intermittent problems that have been seen recently.

  1. The tpstream_writing_test in the 16-Aug-2025 overnight running of the regression tests had a "failed to insert into LB" message in one of its log files.

    • Here is a copy of the message: There were 1 failures to insert data into the latency buffer out of 4887 attempts in the latest monitoring interval.
    • Here is a link to the GitHub CI log: link
  2. The example_system_test in the 09-Aug-2025 overnight running of the regression tests had a "tp_input_2000: Unable to push within timeout period" message in one of its log files (link).

    • I've seen this problem on other regression tests.
    • When I get a chance, I'd like to propose some tests with mutex locking in our "queue" code...
    • This was also seen in the 3ru_1df test on 12-Sep-2025 (link)
    • And in the DAPHNE_TPG part of the readout_type_scan_test on 15-Sep-2025 (link)
    • And here
    • And in the 3ru_1df test on 11-Oct-2025 (link)
  3. I've seen the number of TriggerRecords produced in the example_system_test occasionally fluctuate slightly too low.

    • That happened in the 08-Aug-2025 overnight running of the regression tests. (link)
    • I saw it in a running of all regression tests using dbt-build --integtest on the fddaq-v5.4.0-rc1 candidate release
  4. The DAPHNE_TPG part of the readout_type_scan_test complained about an empty TP fragment in the 04-Sep-2025 overnight running of the regression tests.

    • Here is a copy of the error message: Trigger Primitive fragment for SrcID Trigger_0x00000064 in record (5, 0) has size 72 (outside range [96, 4392])

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the linked GitHub CI logs for tpstream_writing_test, example_system_test, 3ru_1df, and readout_type_scan_test, then compare their reported failures across runs. Done means identifying reproducible causes or contributing conditions for the intermittent insert, timeout, TriggerRecord-count, and empty-fragment failures, with clear follow-up scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.