facebook / facebook/rocksdb

Snapshot management scales poorly

Open
#5,083 18 comments 0 reactions 1 assignee Claimed by @riversand963 View on GitHub
performance
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

`GetSnapshotImpl` and `ReleaseSnapshot` acquire the global `mutex_` to perform their operations. This can result in high contention and therefore bad scalability for workloads that make heavy use of snapshots. This can be seen by running the "randomwithverify" benchmark (this seems to be the only one that uses snapshots).

This is a screenshot from a VTune analysis of the "randomwithverify" benchmark with 4 threads. It clearly shows how the snapshot operations are serialized (the yellow lines indicate transitions where a mutex unlock wakes another thread that was waiting for it).
![image](https://user-images.githubusercontent.com/8653812/54590154-b1485b80-4a27-11e9-84f9-56d7821a10ca.png)
Obviously, a higher number of threads intensifies this problem...

I have considered different ways to improve the situation:
* use a lock-free doubly-linked list for the snapshot list (e.g., based on [Sundell and Tsigas](https://arxiv.org/pdf/cs/0408016.pdf)).
* use _N_ separate snapshot lists, each protected by its own mutex, and distribute the operating threads evenly among them.
* if the mutex is already locked, record the operation and let the mutex-owning thread apply the recorded operations (similar to [flat combining](https://www.cs.bgu.ac.il/~hendlerd/papers/flat-combining.pdf)).

But obviously they all have their own pros and cons, so I would like to get some input from the core team.

I suppose you are aware of this potential bottleneck - do you have other ideas or even plans to improve scalability of snapshots? If we can agree on how to approach this, I am happy to make the according changes and create a PR.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.