cockroachdb / cockroachdb/pebble

experiment: Evaluate recent research on optimizing LSM-trees using AI/ML techniques

Open
#5,058 0 comments 0 reactions 0 assignees View on GitHub
A-storage C-enhancement C-performance T-storage
Dominant language
Go
Stars
6k
Forks
584
Avg merge
16h 35m
Merged PRs (30d)
5

Description

Recent academic research explores AI and machine-learning-based approaches to optimize the performance and efficiency of LSM-tree storage engines. We should allocate time to carefully evaluate these approaches, prototype promising ideas, and determine their potential application to our storage engine.

Relevant Papers:

- [CAMAL: Optimizing LSM-trees via Active Learning](https://arxiv.org/pdf/2409.15130)
- [Leaper: A Learned Prefetcher for Cache Invalidation in LSM-tree based Storage Engines](https://www.vldb.org/pvldb/vol13/p1976-yang.pdf)
- [Learning to Optimize LSM-trees: Towards A Reinforcement Learning based Key-Value Store for Dynamic Workloads](https://arxiv.org/pdf/2308.07013)

Goals:
- Summarize key concepts and proposed benefits.
- Prototype and evaluate promising strategies from these papers.
- Provide recommendations on potential integration.

Jira issue: PEBBLE-1086

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the three linked papers: CAMAL, Leaper, and Learning to Optimize LSM-trees. Summarize their key concepts and benefits, prototype and evaluate promising strategies in the Pebble storage engine, and finish with recommendations for potential integration.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, machine-learning
Domain
databases, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.