facebook / facebook/rocksdb

Using malloc_usable_size() in arena triggers compactions early and results in smaller files and diff number of files

Open
#4,252 6 comments 0 reactions 1 assignee Claimed by @anand1976 View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

> Note: Please use Issues only for bug reports. For questions, discussions, feature requests, etc. post to dev group: https://www.facebook.com/groups/rocksdb.dev

### Expected behavior

Some tests such as column_family_test,db_compaction_test rely on producing a specific number of SST files in directories. To that end they employ a `SpecialMemTable` rep that causes flushes to take place once once a certain threshold is reached.

### Actual behavior

However, the compaction is triggered based on the amount of memory used within the `Memtable`. Some debug and non-debug allocators have a higher allocation overhead and the `malloc_usable_size` reported is higher. This results in earlier flushes, smaller files, smaller sizes per level or db path.

Some tests such as `db_compaction_test` and `column_family_test` expect certain number of files to be created as a result of inserting data. This creates a erroneous connection between the amount data inserted and the amount of memory allocated within the `Memtable`, and this results in smaller amount of data inserted per file, the number of files is higher and the tests fail.

IMHO, either the number

### Steps to reproduce the behavior

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.