clockworklabs / clockworklabs/SpacetimeDB
Don't add committed rows to freshly created indexes
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 25.2k
- Forks
- 1.1k
- Avg merge
- 2d 7h
- Merged PRs (30d)
- 46
Description
The FIXME here:
is a problem.
This came up while trying to implement #1523. 1523 is just a convenience, so this isn't a huge deal that it's blocked. However, this is going to be a much bigger problem for implementing spacetime migrate.
The issue is that, when we create an index during the transaction, we populate the index with rows in the committed state, as well as rows in the transaction state. This makes it impossible to read the row corresponding to an index entry in general! The row could be in one of two places, the committed state, or the transaction state. But this bit is not stored anywhere.
This is fixed once the transaction is committed -- the transaction state is merged into the committed state and the index is repopulated:
It's only an issue when performing queries during a transaction that added an index. This will mainly come up during migrations.
cc @gefjon
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the FIXME in crates/core/src/db/datastore/locking_tx_datastore/mut_tx.rs at lines 365-371, then compare it with index repopulation in committed_state.rs at lines 517-523. Trace how indexes created during a transaction are populated and how committed and transaction rows are represented. Done means queries during a transaction can reliably resolve entries in a newly created index without mixing committed and transaction state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100