clockworklabs / clockworklabs/SpacetimeDB

Confirmed reads: updates lost during module swap due to race with websocket close

Open
#4,417 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
25.2k
Forks
1.1k
Avg merge
2d 7h
Merged PRs (30d)
46

Description

Summary

Clients using confirmed reads (?confirmed=true) can lose subscription updates during a module update that triggers DisconnectAllUsers. The deletion/update messages are stuck waiting for durability confirmation while the websocket close handshake races ahead and kills the connection.

Root Cause

In crates/client-api/src/routes/subscribe.rs, ws_send_loop_inner uses a biased tokio::select! where control messages (unordered) always preempt data messages (messages):

tokio::select! {
    biased;
    maybe_msg = unordered.recv() => { /* Close/Ping/errors */ }
    n = frames_rx.recv_many(...), if !closed => { /* encoded frames */ }
    Some(message) = messages.recv(), if !closed => { /* ClientConnectionReceiver */ }
}

During a module swap (UpdatePerformedWithClientDisconnect):

  1. Migration tx commits, commit_and_broadcast_event queues deletion updates to subscribers with a tx_offset
  2. ClientConnectionReceiver::recv() picks up the deletion, sees confirmed_reads = true, blocks on durable.wait_for(tx_offset) (waiting for fsync)
  3. Host controller replaces the module watcher, disconnects clients, exits old module, drops old watch::Sender<ModuleHost>
  4. watch_module_host returns Err(NoSuchModule), ws_main_loop sends a Close frame via unordered_tx
  5. Biased select picks up the Close frame before messages.recv() completes
  6. Close processing sets closed = true and calls messages.close(), permanently disabling the messages.recv() branch
  7. The deletion update stuck in durable.wait_for() is never delivered

ClientConnectionReceiver::recv() is cancel-safe (stores the pending message in self.current), but it does not matter because the message is never polled again after closed = true.

Reproduction

In crates/smoketests/tests/auto_migration.rs, test_add_table_columns:

  • With confirmed=false (current default in the test): passes
  • With confirmed=true: subscribers fail to receive the expected deletion update

Possible Fixes

  1. Before sending the Close frame in ws_send_loop_inner, drain pending messages from messages (with a short timeout)
  2. When closing due to module exit, temporarily bypass the durability wait (the data is committed, just not yet fsync'd, and the client is about to be disconnected anyway)
  3. Have ClientConnectionReceiver::recv() detect the module-exit condition and stop waiting for durability, returning the message immediately

Impact

Any client using confirmed reads will silently lose the final subscription updates during a module update that triggers client disconnection. This is a data consistency issue since confirmed reads is specifically meant to guarantee that clients only see durable data.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in crates/client-api/src/routes/subscribe.rs, focusing on ws_send_loop_inner and its handling of unordered, messages, and confirmed reads. Then run crates/smoketests/tests/auto_migration.rs test_add_table_columns with confirmed=true to reproduce the lost deletion update. Done means the confirmed subscription receives the expected update during module replacement and the existing confirmed=false case still passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
api, backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.