dragonflydb / dragonflydb/dragonfly
Epic: implement selective tiering for hmap and lists
- Dominant language
- C++
- Stars
- 31.5k
- Forks
- 1.3k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 137
Description
This issue serves as a placeholder for brainstorming the design, surfacing concerns, and outlining potential solutions for selective tiering in hmap.
I prioritized HMAP over other collection types due to its higher priority and relative simplicity. Please refer to the current async tiering design: https://github.com/dragonflydb/dragonfly/blob/main/docs/async-tiering.md
**Goal:** Support selective offloading of hashmap values that are relatively big (> 1KB).
**Non-goal:** Grouping and offloading of multiple small values on the same page.
My early impressions are based on `HMapWrap`, which exposes the minimal required API:
1. Find (return key and value)
2. Return (entire) iterable Range
3. Erase(k)
4. Add(k, v)
5. UpdateExisting(k)
There's also `Get`, which leaks implementation but appears to be used similarly to Range().
**Initial analysis:**
1. Element additions work similarly to Strings and are not problematic, as we write into memory first and then trigger offloading.
2. Deletions are CPU-only operations and do not require I/O .
The major challenge lies in reads: we need to issue multiple reads in a single operation, especially for high-level APIs like HMGET, HGETALL, HGET, HSCAN, HVALS. There is also a challenge of tracking a two-level taxonomy `key` -> `hmap-key` while handling contending IO requests (double-reads, delete-after-read, etc).
But in a way we already have a taxonomy as we support multiple databases.
**Assignees:** @dranikpg (design phase involvement: @romange too)
Contributor guide
Assessment
This issue has not been assessed yet.