HarperFast / HarperFast/harper
Durable MQTT catch-up delivers phantom notification for a TTL-expired message
- Dominant language
- JavaScript
- Stars
- 89
- Forks
- 10
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 200
Description
## Summary
A durable/offline MQTT subscriber's reconnect catch-up can deliver a notification for a message that has **already TTL-expired** — a phantom, stale notification, not just a missed one. This is the sharper opposite of the known, defensible "catch-up silently skips evicted messages" gap: here catch-up actively notifies about content that's already gone.
## Repro
Harper main `ece7da476` (v5.1.15). Chat-with-attachments scenario: `Message @table(expiration:8)`, compound-id topic hierarchy. A durable session goes offline; 5 attachment-bearing messages publish at 2s intervals under an 8s TTL; the session reconnects 3s after the last publish. By reconnect time, `msg0`/`msg1` have genuinely expired (confirmed via oracle: direct table read → 404, SQL count → 0).
**The durable session's audit-log-driven catch-up delivers MQTT notifications for `msg0`/`msg1` anyway.** Dereferencing their attachments afterward is clean (404, no hang/corruption) — so this isn't data loss or a hang. But for a real client it means rendering a message bubble whose attachment immediately 404s on load: a stale/inconsistent notification, not silence.
## Root cause
Catch-up replays from the **audit log** (which still retains the entry after the live record has TTL-expired) without checking whether the live record has since expired before notifying the reconnecting subscriber.
## Distinguishes from D-021 (known, doc-only note)
D-021 (from QA-040) described catch-up **silently skipping** evicted messages entirely — a defensible, documentable gap (subscriber legitimately never learns about something that expired while it was offline). This finding is the opposite and a sharper failure: catch-up **actively notifies** about content that's already gone, rather than staying silent about it.
## Suggested fix direction
Either have catch-up skip audit entries whose live record has since expired (bringing this in line with D-021's already-accepted behavior), or give clients a way to detect staleness from the notification itself (e.g. a flag/marker when the underlying record is already gone by delivery time).
Test file (not yet promoted to the permanent suite): `integrationTests/qa-scratch/qa499-chat-attachments.test.ts`.
— KrAIs 🤖 (exploratory QA, on Kris's behalf)
Contributor guide
Research direction
Start with integrationTests/qa-scratch/qa499-chat-attachments.test.ts and reproduce the durable-session reconnect scenario described in the issue. Trace the audit-log-driven catch-up path and its relationship to the expired live record; done means catch-up no longer notifies for msg0/msg1 after TTL expiry, with a permanent regression test covering the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, nodejs
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100