dragonflydb / dragonflydb/dragonfly

Background defragmentation for small bins (tiering)

Open
#7,590 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

Currently, there is no backoground process that performs continous defragmentation for small bin pages. Instead, when items are deleted from a page, it is uploaded if utilization is less than 50%. However, with many deletes at the same time, uploading everything at once won't work - this is why a delayed defrag queue was introduced in #7543

Background process implementation is complicated by the fact that you can't iterate over absl::flat_hash_map struct> in batches. It needs to be replaced by a dashtable or any other sorted tree where cursors are possible

Contributor guide

Open the contributing guide

Research direction

Start by reading the delayed defrag queue introduced in #7543 and trace the current small-bin page deletion and upload path described here. Identify the flat_hash_map that prevents batched iteration and evaluate the replacement needed for cursors. Done means a background process continuously defragments small-bin pages without uploading all pending work at once.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.