clockworklabs / clockworklabs/SpacetimeDB

Don't add committed rows to freshly created indexes

Open
#1,525 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug release-1.0
Dominant language
Rust
Stars
25.2k
Forks
1.1k
Avg merge
2d 7h
Merged PRs (30d)
46

Description

The FIXME here:

https://github.com/clockworklabs/SpacetimeDB/blob/3e6f91be8969b2f5ceb3beed60434de3393252c4/crates/core/src/db/datastore/locking_tx_datastore/mut_tx.rs#L365-L371

is a problem.

This came up while trying to implement #1523. 1523 is just a convenience, so this isn't a huge deal that it's blocked. However, this is going to be a much bigger problem for implementing spacetime migrate.

The issue is that, when we create an index during the transaction, we populate the index with rows in the committed state, as well as rows in the transaction state. This makes it impossible to read the row corresponding to an index entry in general! The row could be in one of two places, the committed state, or the transaction state. But this bit is not stored anywhere.

This is fixed once the transaction is committed -- the transaction state is merged into the committed state and the index is repopulated:

https://github.com/clockworklabs/SpacetimeDB/blob/3e6f91be8969b2f5ceb3beed60434de3393252c4/crates/core/src/db/datastore/locking_tx_datastore/committed_state.rs#L517-L523

It's only an issue when performing queries during a transaction that added an index. This will mainly come up during migrations.

cc @gefjon

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the FIXME in crates/core/src/db/datastore/locking_tx_datastore/mut_tx.rs at lines 365-371, then compare it with index repopulation in committed_state.rs at lines 517-523. Trace how indexes created during a transaction are populated and how committed and transaction rows are represented. Done means queries during a transaction can reliably resolve entries in a newly created index without mixing committed and transaction state.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.