dragonflydb / dragonflydb/dragonfly
Read Skew for multishard reads transactions
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
Imagine the following:
1. two shards A, B with two keys A, B
2. a `MGET A B` that runs in a loop
3. an `INCR A` and `INCR B` running one after another in a loop
Both (2) and (3) run on different pool threads (lets name them T2 and T3). T3 owns shard A, T2 owns shard B
One would assume that MGET A B always reads A >= B
However:
1. `MGET A B` starts running, submits to shard A and now the thread is taken off by the system scheduler (it couldn't yet submit to shard B)
2. T3 processes `MGET A` - it runs optimistically and the commited txid is not touched, they keys are unlocked - not a tace of it running is left
3. T3 processes step (3) - first, `INCR A` runs inline, then it runs `INCR B` which submits to shard B
4. T2 wakes up and submits `MGET B` to shard B. It runs `INCR B` and then `MGET B`
So the order becomes
```
A B
MGET A
INCR A
INCR B
MGET B
```
See #6215
See #7489
Contributor guide
Research direction
Start by reading the read-skew scenario in this issue and the referenced issues #6215 and #7489. Reproduce the interleaving with MGET A B, INCR A, and INCR B across two shards; done means multishard reads no longer observe A and B from inconsistent transaction points.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100