questdb / questdb/questdb

QWP durable ack: stale per-table watermark survives same-connection drop/recreate

Open
#7,496 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug Core QWP
Dominant language
Java
Stars
17.3k
Forks
1.6k
Avg merge
5d 10h
Merged PRs (30d)
28

Description

Summary

In QWP durable-ack (store-and-forward-with-durability) mode, a client's per-table
durable-upload watermark survives a same-connection drop/recreate (or rename +
recreate) under a stable table name
, so a stale-high watermark from the old
incarnation falsely covers the new incarnation's low seqTxns. The client can then
trim its store-and-forward copy of the new incarnation's rows before the server has
confirmed their object-store upload -- a durability promise the server did not make
for that WAL segment.

This is distinct from #7482. #7482 is a server under-report (the threshold commit
in QwpWalAppender.appendToWalColumnar never feeds recordCommittedTable, so the
txn is never registered). This one is a watermark-staleness mechanism that fires
even when the server reports correctly: the wire durable-ack is name-keyed and the
client merges it monotonically, with no signal to reset a per-table watermark when
the table is re-incarnated under the same name mid-connection.

Mechanism

Client (java-questdb-client,
.../qwp/client/sf/cursor/CursorWebSocketSendLoop.java):

  • durableTableWatermarks is a per-table (name -> seqTxn) map, "Reset on every
    reconnect ... holding stale watermarks across the wire boundary would falsely
    advance trim before re-confirmation" (field comment). Within a live connection it
    is only ever advanced (max): an incoming STATUS_DURABLE_ACK entry updates a table
    only when the new seqTxn is greater, because the wire contract is "monotonically
    non-decreasing per table".
  • A pending OK-acked batch pops (and its SF slot is trimmed) once every
    (name, seqTxn) it carries is <= durableTableWatermarks[name].

Server (QwpIngressProcessorState):

  • collectDurableProgress reports registry.getDurablyUploadedSeqTxn(dirName) keyed
    by the current dir name, so after a drop/recreate it reports the new, low
    incarnation's upload seqTxn.
  • recordCommittedTable resets lastDurableSeqTxns[name] = -1 when it sees the dir
    name change, so the server re-reports that low value.

The two do not cooperate: the server re-emits a decreasing per-table durable
seqTxn on re-incarnation, which violates the "monotonically non-decreasing per table"
contract the client relies on; the client's max-merge silently ignores the decrease
and keeps the old-incarnation watermark. New-incarnation batches whose seqTxns fall
under that stale watermark are treated as durable and trimmed immediately on OK-ack,
without waiting for their own upload.

Trigger

Durable-ack mode + drop/recreate (or rename-away + recreate) churn under a stable
table name on one long-lived connection -- the same churn class #7383 targets for the
WAL-writer cache. The old incarnation must have advanced its durable watermark (e.g.
to N) before the drop; the new incarnation then starts at a low seqTxn and its rows
are covered by N.

Consequence

Durable-ack-grade data loss: the client discards its replayable copy of committed-but-
not-yet-uploaded rows on a durability guarantee that only held for the previous
incarnation of the name. Non-durable (plain OK-ack) mode is unaffected -- an OK there
means WAL-committed, which is true for the new incarnation's rows.

Relationship to #7383 / #7482

  • Pre-existing; not introduced or worsened by #7383. #7383's rename+recreate
    salvage adds one new trigger shape (a salvage records a txn under the renamed
    name), but that path is at-least-once duplicate delivery, not loss.
  • Distinct mechanism from #7482 (server under-report vs client watermark staleness).
    Both are "durable ack covers WAL data the uploader has not confirmed"; recommend
    tracking together.

Possible fix

Carry an incarnation discriminator on the wire so a per-table durable watermark can be
reset within a connection when the table is re-incarnated -- e.g. include the table
dir name (or a monotonic epoch) in the durable-ack entries and have the client reset
durableTableWatermarks[name] when the discriminator changes, instead of relying on
whole-connection reset at reconnect. Alternatively, have the server force a watermark
reset signal (or sever the durable-ack connection) when it observes a same-name dir
change. Requires a coordinated server + pinned-client change and a pinned-client
durable-ack regression test.

Surfaced by the level-3 review of #7383 (adversarial cross-context pass); the
load-bearing client behaviour was confirmed against the currently pinned client.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with java-questdb-client's CursorWebSocketSendLoop.java and the server-side QwpIngressProcessorState, tracing durableTableWatermarks, collectDurableProgress, and recordCommittedTable across a same-name drop/recreate. Review the durable-ack wire contract and the pinned-client durable-ack regression test requirement. Done means a new incarnation cannot inherit the old watermark and the regression test prevents premature store-and-forward trimming.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.