mafintosh / mafintosh/rollbackdb
a readstream of a checkout might iterate a lot of unnecessary data
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 40
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Our history is stored as an append-only change log, and for the purposes here you can imagine them as incrementing numbers, so if you do `put foo = hello` then `put foo = goodbye` you would have changes `1` and `2` in the log.
We want to support accessing data in a key/value store at the state it was relative to a specific point in time in the history. We call this feature 'checkouts', meaning you can checkout the state of the db to a specific change #. So if you do `get foo` at checkout `1`, it looks for the most recent version of `foo` that appeared in the history before change #1 and returns that.
Currently data is stored like this:
```
key!change => value
```
And read streams work like this
1. Create new iterator
2. `iterate.next()` to find next key
3. seek to `key!checkout` and do `iterate.prev()` to find right version
4. seek to `key!~` and goto 2.
We take advantage of a LevelDB feature that lets us seek the iterator to the next matching key in the lexicographically sorted keyspace in an efficient way, but we have to know the key want to skip to.
The smart thing about this setup is that we can find the right version of key really fast when doing a simple `get(key)` since that only requires one seek.
The downside is that if a new version of the database adds a bunch of keys between two keys of an old version, then doing a read stream of the old version of the database is gonna take longer since we need to seek through these keys
For example, lets say we have a database with two keys, `alice` and `zoey` that were both added a version `1`. We then in version `2` add `bob`, `cat` and `dog`.
The database now looks like this
```
alice!1
bob!2
cat!2
dog!2
zoey!1
```
If we do a read stream checkout at `1` we now need to scan through `bob`, `cat`, `dog` to get to `zoey` (which is the value we are looking for)
Any ideas on how to optimize this would be appreciated :)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the read-stream iterator described in the issue and reviewing the key!change layout and LevelDB seek behavior. Reproduce the checkout example with alice, zoey, bob, cat, and dog, then define an optimization that preserves checkout results while avoiding unnecessary scans; no source file or test is identified in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100