Usage of memory is too high when use default malloc of Ubuntu
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev
### Expected behavior
Usage of memory is normal on both malloc and jemalloc
### Actual behavior
The usage of memory is 25GB when use default malloc, but is 13GB when use jemalloc.
Memory statistics: block cache: 8.5GB, memtable: 1.5GB, filter/index: 1GB, pinned memory : < 1MB
It seems usage of memory is only normal on jemalloc.
### Steps to reproduce the behavior
OS: Linux version 4.15.0-45-generic (buildd@lgw01-amd64-031) (gcc version 7.3.0 (Ubuntu 7.3.0-16ubuntu3))
Rocksdb version 5.18.3
Data size is 93G with lz4 compression
config:
```
options.create_if_missing = true;
options.create_missing_column_families = true;
options.db_write_buffer_size = 1024 * 1024 * 1024;
options.max_background_jobs = 8;
options.max_subcompactions = 4;
options.bytes_per_sync = 1 * 1024 * 1024;
options.compaction_pri = rocksdb::kMinOverlappingRatio;
options.sst_file_manager= rocksdb_sst_file_manager;
column_options.compression = rocksdb::CompressionType::kLZ4Compression;
column_options.prefix_extractor.reset(rocksdb::NewFixedPrefixTransform(3));
column_options.memtable_prefix_bloom_size_ratio = 0.02;
column_options.max_write_buffer_number = 6;
column_options.min_write_buffer_number_to_merge = 2;
column_options.write_buffer_size = 128 * 1024 * 1024;
column_options.max_bytes_for_level_base = 1024 * 1024 * 1024;
column_options.level_compaction_dynamic_level_bytes = true;
column_options.target_file_size_base = 128 * 1024 * 1024;
table_options.block_cache = cache; //8G
table_options.index_type = rocksdb::BlockBasedTableOptions::IndexType::kHashSearch;
table_options.data_block_index_type = rocksdb::BlockBasedTableOptions::DataBlockIndexType::kDataBlockBinaryAndHash;
table_options.filter_policy.reset(rocksdb::NewBloomFilterPolicy(10, false));
table_options.format_version = 4;
table_options.block_size = 16 * 1024;
```
Contributor guide
Assessment
This issue has not been assessed yet.