Snapshot management scales poorly
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
`GetSnapshotImpl` and `ReleaseSnapshot` acquire the global `mutex_` to perform their operations. This can result in high contention and therefore bad scalability for workloads that make heavy use of snapshots. This can be seen by running the "randomwithverify" benchmark (this seems to be the only one that uses snapshots).
This is a screenshot from a VTune analysis of the "randomwithverify" benchmark with 4 threads. It clearly shows how the snapshot operations are serialized (the yellow lines indicate transitions where a mutex unlock wakes another thread that was waiting for it).

Obviously, a higher number of threads intensifies this problem...
I have considered different ways to improve the situation:
* use a lock-free doubly-linked list for the snapshot list (e.g., based on [Sundell and Tsigas](https://arxiv.org/pdf/cs/0408016.pdf)).
* use _N_ separate snapshot lists, each protected by its own mutex, and distribute the operating threads evenly among them.
* if the mutex is already locked, record the operation and let the mutex-owning thread apply the recorded operations (similar to [flat combining](https://www.cs.bgu.ac.il/~hendlerd/papers/flat-combining.pdf)).
But obviously they all have their own pros and cons, so I would like to get some input from the core team.
I suppose you are aware of this potential bottleneck - do you have other ideas or even plans to improve scalability of snapshots? If we can agree on how to approach this, I am happy to make the according changes and create a PR.
Contributor guide
Assessment
This issue has not been assessed yet.