RocksDB occupied massive memory and going up
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
I calculated memory usage in worst case is about 100GB, but actually it occupied approximately 230GB.
I checked process memory using pmap, it mapped massive sst files to resident memory, so it occupied 194GB approximately, and the configuration of `max_open_files=20000`.
How to limit rocksdb maps sst file to memory?
### Expected behavior
I have caculated max memory usage of rocksdb is about 100GB, but it occupied 230GB.
### Actual behavior
pmap the process of rocksdb, found massive sst files were mapping to resident memory, about 3200 sst files(194GB)
```
7f2ae3edc000 ---p 00000000 00:00 0 1168 0 0 0 0 0 0
7f2ae4000000 rw-p 00000000 00:00 0 64604 64604 64604 64604 64604 0 0
7f2ae7f17000 ---p 00000000 00:00 0 932 0 0 0 0 0 0
7f2ae8000000 rw-p 00000000 00:00 0 65444 65444 65444 65444 65444 0 0
7f2aebfe9000 ---p 00000000 00:00 0 92 0 0 0 0 0 0
7f2aec000000 rw-p 00000000 00:00 0 63148 63148 63148 63148 63148 0 0
```
the rocksdb's configuration of `max_open_files=20000`, i have checked rocksdb opened 20000 sst files approximately.
memory usage i caculated:
- table cache : 90GB
- Block cache : 8MB
- Memtables : 320MB
- Blocks pinned by iterators : 100MB
**table cache**
```
min(max_open_files, file_num) * (index_block_size + filter_block_size)
```
```
./sst_dump --file=1524151.sst --command=noop --show_properties
data block size: 67109625
index block size (user-key? 0, delta-value? 0): 577149
filter block size: 4286405
```
So, table cache is : `min(20000, 31000) * (577149 + 4286405) = 90GB`
**memtable**
```
max_write_buffer_number=5
write_buffer_size=67108864
```
So, memtable is :
`max_write_buffer_number * write_buffer_size = 320MB`
**Blocks pinned by iterators**
worst case is about 100MB
mmap options are default
```
// If true, then use mmap to read data
bool use_mmap_reads = false;
// If true, then use mmap to write data
bool use_mmap_writes = true;
```
### Steps to reproduce the behavior
Contributor guide
Assessment
This issue has not been assessed yet.