facebook / facebook/rocksdb

Long lived Iterator pinning flushed MemTable and compacted SSTs

Open
#10,536 3 comments 0 reactions 0 assignees View on GitHub
design discussion up-for-grabs
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

### Expected behavior
Flushed **MemTable**s and compacted **SST**s should be deleted.

### Actual behavior
Long lived Iterators referring `Version` objects which referring **MemTable**s and compacted **SST**s which lead to memory over consumption while writing to DB.

### Steps to reproduce the behavior
Create an iterator and using it while writing to DB, newly created MemTables can not be freed.

------

We noticed this issue when we running sysbench on myrocks, we also have filed an [issue](https://github.com/facebook/mysql-5.6/issues/1211) and [PR](https://github.com/facebook/mysql-5.6/pull/1212) for MyRocks, we worked around this issue by creating a new Iterator and delete old Iterator periodically. It is better to resolve this issue in RocksDB.

Contributor guide

Open the contributing guide

Research direction

The report names long-lived Iterators, Version objects, MemTables, and compacted SSTs but does not identify files or tests. Start by reproducing the issue while writing to the database with an iterator held open, then trace the referenced object lifetimes and add or run a regression test showing that obsolete flushed MemTables and compacted SSTs are released without requiring periodic iterator replacement.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.