dragonflydb / dragonflydb/dragonfly

Epic: implement selective tiering for hmap and lists

Open
#6,447 6 comments 0 reactions 1 assignee Claimed by @dranikpg View on GitHub
Dominant language
C++
Stars
31.5k
Forks
1.3k
Avg merge
1d 10h
Merged PRs (30d)
137

Description

This issue serves as a placeholder for brainstorming the design, surfacing concerns, and outlining potential solutions for selective tiering in hmap.

I prioritized HMAP over other collection types due to its higher priority and relative simplicity. Please refer to the current async tiering design: https://github.com/dragonflydb/dragonfly/blob/main/docs/async-tiering.md

**Goal:** Support selective offloading of hashmap values that are relatively big (> 1KB).
**Non-goal:** Grouping and offloading of multiple small values on the same page.

My early impressions are based on `HMapWrap`, which exposes the minimal required API:

1. Find (return key and value)
2. Return (entire) iterable Range
3. Erase(k)
4. Add(k, v)
5. UpdateExisting(k)

There's also `Get`, which leaks implementation but appears to be used similarly to Range().

**Initial analysis:**
1. Element additions work similarly to Strings and are not problematic, as we write into memory first and then trigger offloading.
2. Deletions are CPU-only operations and do not require I/O .

The major challenge lies in reads: we need to issue multiple reads in a single operation, especially for high-level APIs like HMGET, HGETALL, HGET, HSCAN, HVALS. There is also a challenge of tracking a two-level taxonomy `key` -> `hmap-key` while handling contending IO requests (double-reads, delete-after-read, etc).

But in a way we already have a taxonomy as we support multiple databases.

**Assignees:** @dranikpg (design phase involvement: @romange too)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.