dragonflydb / dragonflydb/dragonfly
Two-step asynchronous IO in SliceSnapshot
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
Currently RestoreStreamer uses `AsyncWrite` with a custom backpressure mechanism tracking the in-flight bytes. SliceSnapshot, on the other hand, uses the buffer from RdbSerializer and fiber-based socket flushes when it fills up.
Transitioning to async io has the following benefits:
1. Better memory tracking - precise tracking of in-flight bytes and used memory with a global cap that can be respected
2. Evened out backpressure - backpressure is applied equally to all clients and can be applied before a client blocks with its own buffer
3. Two step write and backpressure - already possible now with the intermediate buffer
Contributor guide
Research direction
Start by reading SliceSnapshot and its use of the RdbSerializer buffer, then compare it with RestoreStreamer, AsyncWrite, and the existing fiber-based socket flushes. Trace how in-flight bytes, buffers, and backpressure are currently handled. Done means SliceSnapshot uses the proposed two-step asynchronous write and backpressure flow while respecting global memory tracking and client-level backpressure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100