HarperFast / HarperFast/harper

Flaky unit test: MQTT 'subscribe with QoS=1 and reconnect with non-clean session' silent 20s timeout (lmdb pass)

Open
#2,274 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
89
Forks
10
Avg merge
2d 6h
Merged PRs (30d)
200

Description

## Summary

`test MQTT connections and commands > subscribe with QoS=1 and reconnect with non-clean session` (unitTests/apiTests/mqtt-test.mjs) timed out at its 20s mocha limit on `main`, taking the Unit Test workflow red at head-of-main. Filed to track the flake behind its quarantine (the test is skipped with a pointer here until this is root-caused).

Evidence run: [Unit Test run 32531526557, job "Unit Test (Node.js v26)"](https://github.com/HarperFast/harper/actions/runs/32531526557) on `main` (sha f8a5aa90), 2026-08-21.

## What the CI log shows

- The failure happened in the **lmdb** pass (`test:unit:lmdb` → `HARPER_STORAGE_ENGINE=lmdb npm run test:unit:apitests`), not the default rocksdb pass — the same test had passed the rocksdb pass minutes earlier in the same job (62ms).
- The test's `beforeEach` ran (its four `clear() is deprecated` lines appear at 22:12:28.26), then **nothing** for 20.15s until mocha's `Error: Timeout of 20000ms exceeded` at 22:12:48.41. No `[mqtt]` server-side log lines in between.
- The failure is a silent hang, not an assertion: the test's own final wait (the 3-queued-messages check) has a 15s bounded reject with a descriptive message that never fired, so the hang was in one of the earlier steps that have **no individual timeout**: one of the three `connectAsync` calls, `subscribeAsync`, one of the three `endDurableSession` waits (each awaits a server-side `'disconnected'` event), the QoS-1 publish/acknowledge wait (awaits a server-side `'acknowledged'` event), or one of the three `publishAsync` (PUBACK) waits.

## Local reproduction attempt

Not reproduced: 30 consecutive full runs of `mqtt-test.mjs` under `HARPER_STORAGE_ENGINE=lmdb`, pinned to 2 cores (with unrelated suites loading the other cores) on Linux/Node 26 — 0 failures. Single-test runs also pass consistently.

## Hypotheses worth checking when this is picked up

1. **Reconnect racing prior session teardown.** The test repeatedly does `endDurableSession` (which resolves on the server's `'disconnected'` event) and immediately reconnects with the same clientId (`test-client1`, `clean: false`). If `'disconnected'` is emitted before the durable session's async teardown/persistence completes, the next CONNECT for the same clientId races the takeover and could stall before CONNACK.
2. **QoS downgrade on delivery.** The publish/acknowledge step resolves only when the server emits `'acknowledged'` for the client's PUBACK. If the delivery path ever hands the message over at QoS 0 (e.g. via the current-state/queued-delivery path rather than the live listener), no PUBACK ever arrives and the await hangs silently.
3. lmdb-specific timing in durable-session persistence (the failure occurred only in the lmdb pass so far — one observation, so weak evidence).

## Suggested reinstatement path

Wrap each unbounded await in the test with a labeled bounded timeout (per-step `AbortSignal.timeout`/`Promise.race` with a descriptive error) so the next occurrence pinpoints the hanging step, then unskip and let CI collect evidence.

Contributor guide

Open the contributing guide

Research direction

Start with unitTests/apiTests/mqtt-test.mjs and run the named MQTT test under HARPER_STORAGE_ENGINE=lmdb. Inspect each unbounded connect, subscribe, session-ending, publish, and acknowledgement wait, using the CI run and listed hypotheses as context. Done means every step has a labeled bounded timeout, the test is reinstated, and CI provides more specific failure evidence.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.