clockworklabs / clockworklabs/SpacetimeDB
Confirmed reads: updates lost during module swap due to race with websocket close
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 25.2k
- Forks
- 1.1k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 46
Description
Summary
Clients using confirmed reads (?confirmed=true) can lose subscription updates during a module update that triggers DisconnectAllUsers. The deletion/update messages are stuck waiting for durability confirmation while the websocket close handshake races ahead and kills the connection.
Root Cause
In crates/client-api/src/routes/subscribe.rs, ws_send_loop_inner uses a biased tokio::select! where control messages (unordered) always preempt data messages (messages):
tokio::select! {
biased;
maybe_msg = unordered.recv() => { /* Close/Ping/errors */ }
n = frames_rx.recv_many(...), if !closed => { /* encoded frames */ }
Some(message) = messages.recv(), if !closed => { /* ClientConnectionReceiver */ }
}
During a module swap (UpdatePerformedWithClientDisconnect):
- Migration tx commits,
commit_and_broadcast_eventqueues deletion updates to subscribers with atx_offset ClientConnectionReceiver::recv()picks up the deletion, seesconfirmed_reads = true, blocks ondurable.wait_for(tx_offset)(waiting for fsync)- Host controller replaces the module watcher, disconnects clients, exits old module, drops old
watch::Sender<ModuleHost> watch_module_hostreturnsErr(NoSuchModule),ws_main_loopsends aCloseframe viaunordered_tx- Biased select picks up the Close frame before
messages.recv()completes - Close processing sets
closed = trueand callsmessages.close(), permanently disabling themessages.recv()branch - The deletion update stuck in
durable.wait_for()is never delivered
ClientConnectionReceiver::recv() is cancel-safe (stores the pending message in self.current), but it does not matter because the message is never polled again after closed = true.
Reproduction
In crates/smoketests/tests/auto_migration.rs, test_add_table_columns:
- With
confirmed=false(current default in the test): passes - With
confirmed=true: subscribers fail to receive the expected deletion update
Possible Fixes
- Before sending the Close frame in
ws_send_loop_inner, drain pending messages frommessages(with a short timeout) - When closing due to module exit, temporarily bypass the durability wait (the data is committed, just not yet fsync'd, and the client is about to be disconnected anyway)
- Have
ClientConnectionReceiver::recv()detect the module-exit condition and stop waiting for durability, returning the message immediately
Impact
Any client using confirmed reads will silently lose the final subscription updates during a module update that triggers client disconnection. This is a data consistency issue since confirmed reads is specifically meant to guarantee that clients only see durable data.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in crates/client-api/src/routes/subscribe.rs, focusing on ws_send_loop_inner and its handling of unordered, messages, and confirmed reads. Then run crates/smoketests/tests/auto_migration.rs test_add_table_columns with confirmed=true to reproduce the lost deletion update. Done means the confirmed subscription receives the expected update during module replacement and the existing confirmed=false case still passes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100