facebook / facebook/rocksdb

Transaction DB holds too much memory

Open
#4,333 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev

We have a scenario needs multiple DB instances and also multiple CFs within a DB with transaction support.
Transaction DB keeps **max_write_buffer_number** of memtable even when it is flushed, considering the number of DB instance times the CF count, it's significant of memory consumption.
I know those memtable is held for checking write conflicts, but it doesn't seem to be freed when there is no write for a while.
The work around for us is setting **max_write_buffer_number_to_maintain = 1,** but even so, it still uses a lot of memory.

### Expected behavior
Tx DB should free memtable when there is no write for a while, or at least can be freed by manual flush, so the user can decide to free up the memory when there is no recent write.

### Actual behavior
Tx DB consistently holds max_write_buffer_number_to_maintain of memtables, even there is no recent write. Manual flush doesn't help either.

### Steps to reproduce the behavior

Create TransactionDB with max_write_buffer_number = 3, write_buffer_size=64M. Then write enough data (more than 3 * 64MB). DB holds ~192MB memory forever, even after manual flush. If there is N CF in the DB, then the memory overhead becomes even more significant.

Cc: @yuslepukhin

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.