dragonflydb / dragonflydb/dragonfly

Two-step asynchronous IO in SliceSnapshot

Open
#7,472 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

Currently RestoreStreamer uses `AsyncWrite` with a custom backpressure mechanism tracking the in-flight bytes. SliceSnapshot, on the other hand, uses the buffer from RdbSerializer and fiber-based socket flushes when it fills up.

Transitioning to async io has the following benefits:
1. Better memory tracking - precise tracking of in-flight bytes and used memory with a global cap that can be respected
2. Evened out backpressure - backpressure is applied equally to all clients and can be applied before a client blocks with its own buffer
3. Two step write and backpressure - already possible now with the intermediate buffer

Contributor guide

Open the contributing guide

Research direction

Start by reading SliceSnapshot and its use of the RdbSerializer buffer, then compare it with RestoreStreamer, AsyncWrite, and the existing fiber-based socket flushes. Trace how in-flight bytes, buffers, and backpressure are currently handled. Done means SliceSnapshot uses the proposed two-step asynchronous write and backpressure flow while respecting global memory tracking and client-level backpressure.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.