MetPX / MetPX/sr_insects

flakey broker is triggering a case where retryEmptyBeforeExit doesn't work

Open
#89 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Shell
Stars
0
Forks
1
PR merge metrics
No merged PRs in 30d

Description

I haven't been able to reproduce this locally but on GitHub, we often get flakey broker failures like this:

```
test 23 FAILURE: post test2_f61 (1605) should have the same number of files of sender (1125)
```

This post test2_f61 config has `inflight 40s`, which causes messages to be put into `worklist.failed` because they are too young:

```
2026-06-17 23:12:26,967 [WARNING] sarracenia.flow filter file too young: queueing for retry.
2026-06-17 23:12:26,967 [WARNING] sarracenia.flow filter file too young: queueing for retry.
2026-06-17 23:12:26,967 [WARNING] sarracenia.flow filter file too young: queueing for retry.
2026-06-17 23:12:26,967 [WARNING] sarracenia.flow filter file too young: queueing for retry.
2026-06-17 23:12:26,968 [INFO] sarracenia.flow please_stop asked to stop
2026-06-17 23:12:26,968 [INFO] sarracenia.flow _runHousekeeping on_housekeeping pid: 24980 post/test2_f61.conf instance: 0
2026-06-17 23:12:26,979 [INFO] sarracenia.diskqueue on_housekeeping work_retry_00 Number of messages in retry list 35
2026-06-17 23:12:26,979 [INFO] sarracenia.flowcb.housekeeping.resources on_housekeeping Current cpu_times: user=0.64 system=0.04
2026-06-17 23:12:26,979 [INFO] sarracenia.flowcb.housekeeping.resources on_housekeeping Current mem usage: 82.5MiB, accumulating count (0 or 0/100 so far) before self-setting threshold
2026-06-17 23:12:26,980 [INFO] sarracenia.flowcb.log stats version: 3.03.00rc1, started: now, last_housekeeping: 0.1 seconds ago
2026-06-17 23:12:26,980 [INFO] sarracenia.flowcb.log stats messages received: 0, accepted: 0, rejected: 0 rate accepted: 0.0% or 0.0 m/s
2026-06-17 23:12:26,980 [INFO] sarracenia.flowcb.log stats files transferred: 0 bytes: 0Bytes rate: 0Bytes/sec
2026-06-17 23:12:26,982 [INFO] sarracenia.flow please_stop asked to stop
2026-06-17 23:12:26,982 [INFO] sarracenia.flow _runHousekeeping on_housekeeping pid: 24980 post/test2_f61.conf instance: 0
2026-06-17 23:12:26,982 [INFO] sarracenia.diskqueue on_housekeeping still 22 messages in work_retry_00 list. Resuming retries with /home/runner/.cache/sr3/post/test2_f61/diskqueue_work_retry_00
2026-06-17 23:12:26,983 [INFO] sarracenia.flowcb.housekeeping.resources on_housekeeping Current cpu_times: user=0.64 system=0.04
2026-06-17 23:12:26,983 [INFO] sarracenia.flowcb.housekeeping.resources on_housekeeping Current mem usage: 82.5MiB, accumulating count (0 or 13/100 so far) before self-setting threshold
2026-06-17 23:12:26,983 [INFO] sarracenia.flowcb.log stats version: 3.03.00rc1, started: now, last_housekeeping: 0.0 seconds ago
2026-06-17 23:12:26,983 [INFO] sarracenia.flowcb.log stats messages received: 13, accepted: 0, rejected: 13 rate accepted: 0.0% or 0.0 m/s
2026-06-17 23:12:26,983 [INFO] sarracenia.flowcb.log stats files transferred: 0 bytes: 0Bytes rate: 0Bytes/sec
2026-06-17 23:12:26,984 [INFO] sarracenia.flow close flow/close completed cleanly pid: 24980 post/test2_f61.conf instance: 0
```

But once all the messages are in `worklist.failed`, the post process terminates and doesn't attempt to empty its retry list, even though `retryEmptyBeforeExit` is set to True in the config. There's something broken in the `retryEmptyBeforeExit` logic that is not allowing it to detect that there are retries that need to be processed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with test 23, the post/test2_f61.conf configuration, and the retryEmptyBeforeExit setting; compare the logged worklist.failed and work_retry_00 states during shutdown. Trace the retry termination logic from the flow stop path and verify that the test does not exit while retry messages remain pending.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, python, shell
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.