dragonflydb / dragonflydb/dragonfly

Read Skew for multishard reads transactions

Open
#7,494 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

Imagine the following:
1. two shards A, B with two keys A, B
2. a `MGET A B` that runs in a loop
3. an `INCR A` and `INCR B` running one after another in a loop

Both (2) and (3) run on different pool threads (lets name them T2 and T3). T3 owns shard A, T2 owns shard B

One would assume that MGET A B always reads A >= B

However:

1. `MGET A B` starts running, submits to shard A and now the thread is taken off by the system scheduler (it couldn't yet submit to shard B)
2. T3 processes `MGET A` - it runs optimistically and the commited txid is not touched, they keys are unlocked - not a tace of it running is left
3. T3 processes step (3) - first, `INCR A` runs inline, then it runs `INCR B` which submits to shard B
4. T2 wakes up and submits `MGET B` to shard B. It runs `INCR B` and then `MGET B`

So the order becomes
```
A B
MGET A
INCR A
INCR B
MGET B
```

See #6215
See #7489

Contributor guide

Open the contributing guide

Research direction

Start by reading the read-skew scenario in this issue and the referenced issues #6215 and #7489. Reproduce the interleaving with MGET A B, INCR A, and INCR B across two shards; done means multishard reads no longer observe A and B from inconsistent transaction points.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.