HarperFast / HarperFast/harper
Flaky unit test: MQTT 'subscribe with QoS=1 and reconnect with non-clean session' silent 20s timeout (lmdb pass)
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
`test MQTT connections and commands > subscribe with QoS=1 and reconnect with non-clean session` (unitTests/apiTests/mqtt-test.mjs) timed out at its 20s mocha limit on `main`, taking the Unit Test workflow red at head-of-main. Filed to track the flake behind its quarantine (the test is skipped with a pointer here until this is root-caused).
Evidence run: [Unit Test run 32531526557, job "Unit Test (Node.js v26)"](https://github.com/HarperFast/harper/actions/runs/32531526557) on `main` (sha f8a5aa90), 2026-08-21.
## What the CI log shows
- The failure happened in the **lmdb** pass (`test:unit:lmdb` → `HARPER_STORAGE_ENGINE=lmdb npm run test:unit:apitests`), not the default rocksdb pass — the same test had passed the rocksdb pass minutes earlier in the same job (62ms).
- The test's `beforeEach` ran (its four `clear() is deprecated` lines appear at 22:12:28.26), then **nothing** for 20.15s until mocha's `Error: Timeout of 20000ms exceeded` at 22:12:48.41. No `[mqtt]` server-side log lines in between.
- The failure is a silent hang, not an assertion: the test's own final wait (the 3-queued-messages check) has a 15s bounded reject with a descriptive message that never fired, so the hang was in one of the earlier steps that have **no individual timeout**: one of the three `connectAsync` calls, `subscribeAsync`, one of the three `endDurableSession` waits (each awaits a server-side `'disconnected'` event), the QoS-1 publish/acknowledge wait (awaits a server-side `'acknowledged'` event), or one of the three `publishAsync` (PUBACK) waits.
## Local reproduction attempt
Not reproduced: 30 consecutive full runs of `mqtt-test.mjs` under `HARPER_STORAGE_ENGINE=lmdb`, pinned to 2 cores (with unrelated suites loading the other cores) on Linux/Node 26 — 0 failures. Single-test runs also pass consistently.
## Hypotheses worth checking when this is picked up
1. **Reconnect racing prior session teardown.** The test repeatedly does `endDurableSession` (which resolves on the server's `'disconnected'` event) and immediately reconnects with the same clientId (`test-client1`, `clean: false`). If `'disconnected'` is emitted before the durable session's async teardown/persistence completes, the next CONNECT for the same clientId races the takeover and could stall before CONNACK.
2. **QoS downgrade on delivery.** The publish/acknowledge step resolves only when the server emits `'acknowledged'` for the client's PUBACK. If the delivery path ever hands the message over at QoS 0 (e.g. via the current-state/queued-delivery path rather than the live listener), no PUBACK ever arrives and the await hangs silently.
3. lmdb-specific timing in durable-session persistence (the failure occurred only in the lmdb pass so far — one observation, so weak evidence).
## Suggested reinstatement path
Wrap each unbounded await in the test with a labeled bounded timeout (per-step `AbortSignal.timeout`/`Promise.race` with a descriptive error) so the next occurrence pinpoints the hanging step, then unskip and let CI collect evidence.
Contributor guide
Research direction
Start with unitTests/apiTests/mqtt-test.mjs and run the named MQTT test under HARPER_STORAGE_ENGINE=lmdb. Inspect each unbounded connect, subscribe, session-ending, publish, and acknowledgement wait, using the CI run and listed hypotheses as context. Done means every step has a labeled bounded timeout, the test is reinstated, and CI provides more specific failure evidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100