cockroachdb / cockroachdb/pebble
experiment: Evaluate recent research on optimizing LSM-trees using AI/ML techniques
- Dominant language
- Go
- Stars
- 6k
- Forks
- 584
- Avg merge
- 16h 35m
- Merged PRs (30d)
- 5
Description
Recent academic research explores AI and machine-learning-based approaches to optimize the performance and efficiency of LSM-tree storage engines. We should allocate time to carefully evaluate these approaches, prototype promising ideas, and determine their potential application to our storage engine.
Relevant Papers:
- [CAMAL: Optimizing LSM-trees via Active Learning](https://arxiv.org/pdf/2409.15130)
- [Leaper: A Learned Prefetcher for Cache Invalidation in LSM-tree based Storage Engines](https://www.vldb.org/pvldb/vol13/p1976-yang.pdf)
- [Learning to Optimize LSM-trees: Towards A Reinforcement Learning based Key-Value Store for Dynamic Workloads](https://arxiv.org/pdf/2308.07013)
Goals:
- Summarize key concepts and proposed benefits.
- Prototype and evaluate promising strategies from these papers.
- Provide recommendations on potential integration.
Jira issue: PEBBLE-1086
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the three linked papers: CAMAL, Leaper, and Learning to Optimize LSM-trees. Summarize their key concepts and benefits, prototype and evaluate promising strategies in the Pebble storage engine, and finish with recommendations for potential integration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, machine-learning
- Domain
- databases, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100