dragonflydb / dragonflydb/dragonfly
Background defragmentation for small bins (tiering)
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
Currently, there is no backoground process that performs continous defragmentation for small bin pages. Instead, when items are deleted from a page, it is uploaded if utilization is less than 50%. However, with many deletes at the same time, uploading everything at once won't work - this is why a delayed defrag queue was introduced in #7543
Background process implementation is complicated by the fact that you can't iterate over absl::flat_hash_map struct> in batches. It needs to be replaced by a dashtable or any other sorted tree where cursors are possible
Contributor guide
Research direction
Start by reading the delayed defrag queue introduced in #7543 and trace the current small-bin page deletion and upload path described here. Identify the flat_hash_map that prevents batched iteration and evaluate the replacement needed for cursors. Done means a background process continuously defragments small-bin pages without uploading all pending work at once.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100