QWP durable ack: stale per-table watermark survives same-connection drop/recreate
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 17.3k
- Forks
- 1.6k
- Avg merge
- 5d 10h
- Merged PRs (30d)
- 28
Description
Summary
In QWP durable-ack (store-and-forward-with-durability) mode, a client's per-table
durable-upload watermark survives a same-connection drop/recreate (or rename +
recreate) under a stable table name, so a stale-high watermark from the old
incarnation falsely covers the new incarnation's low seqTxns. The client can then
trim its store-and-forward copy of the new incarnation's rows before the server has
confirmed their object-store upload -- a durability promise the server did not make
for that WAL segment.
This is distinct from #7482. #7482 is a server under-report (the threshold commit
in QwpWalAppender.appendToWalColumnar never feeds recordCommittedTable, so the
txn is never registered). This one is a watermark-staleness mechanism that fires
even when the server reports correctly: the wire durable-ack is name-keyed and the
client merges it monotonically, with no signal to reset a per-table watermark when
the table is re-incarnated under the same name mid-connection.
Mechanism
Client (java-questdb-client,
.../qwp/client/sf/cursor/CursorWebSocketSendLoop.java):
durableTableWatermarksis a per-table (name -> seqTxn) map, "Reset on every
reconnect ... holding stale watermarks across the wire boundary would falsely
advance trim before re-confirmation" (field comment). Within a live connection it
is only ever advanced (max): an incomingSTATUS_DURABLE_ACKentry updates a table
only when the new seqTxn is greater, because the wire contract is "monotonically
non-decreasing per table".- A pending OK-acked batch pops (and its SF slot is trimmed) once every
(name, seqTxn)it carries is<=durableTableWatermarks[name].
Server (QwpIngressProcessorState):
collectDurableProgressreportsregistry.getDurablyUploadedSeqTxn(dirName)keyed
by the current dir name, so after a drop/recreate it reports the new, low
incarnation's upload seqTxn.recordCommittedTableresetslastDurableSeqTxns[name] = -1when it sees the dir
name change, so the server re-reports that low value.
The two do not cooperate: the server re-emits a decreasing per-table durable
seqTxn on re-incarnation, which violates the "monotonically non-decreasing per table"
contract the client relies on; the client's max-merge silently ignores the decrease
and keeps the old-incarnation watermark. New-incarnation batches whose seqTxns fall
under that stale watermark are treated as durable and trimmed immediately on OK-ack,
without waiting for their own upload.
Trigger
Durable-ack mode + drop/recreate (or rename-away + recreate) churn under a stable
table name on one long-lived connection -- the same churn class #7383 targets for the
WAL-writer cache. The old incarnation must have advanced its durable watermark (e.g.
to N) before the drop; the new incarnation then starts at a low seqTxn and its rows
are covered by N.
Consequence
Durable-ack-grade data loss: the client discards its replayable copy of committed-but-
not-yet-uploaded rows on a durability guarantee that only held for the previous
incarnation of the name. Non-durable (plain OK-ack) mode is unaffected -- an OK there
means WAL-committed, which is true for the new incarnation's rows.
Relationship to #7383 / #7482
- Pre-existing; not introduced or worsened by #7383. #7383's rename+recreate
salvage adds one new trigger shape (a salvage records a txn under the renamed
name), but that path is at-least-once duplicate delivery, not loss. - Distinct mechanism from #7482 (server under-report vs client watermark staleness).
Both are "durable ack covers WAL data the uploader has not confirmed"; recommend
tracking together.
Possible fix
Carry an incarnation discriminator on the wire so a per-table durable watermark can be
reset within a connection when the table is re-incarnated -- e.g. include the table
dir name (or a monotonic epoch) in the durable-ack entries and have the client reset
durableTableWatermarks[name] when the discriminator changes, instead of relying on
whole-connection reset at reconnect. Alternatively, have the server force a watermark
reset signal (or sever the durable-ack connection) when it observes a same-name dir
change. Requires a coordinated server + pinned-client change and a pinned-client
durable-ack regression test.
Surfaced by the level-3 review of #7383 (adversarial cross-context pass); the
load-bearing client behaviour was confirmed against the currently pinned client.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with java-questdb-client's CursorWebSocketSendLoop.java and the server-side QwpIngressProcessorState, tracing durableTableWatermarks, collectDurableProgress, and recordCommittedTable across a same-name drop/recreate. Review the durable-ack wire contract and the pinned-client durable-ack regression test requirement. Done means a new incarnation cannot inherit the old watermark and the regression test prevents premature store-and-forward trimming.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100