facebook / facebook/rocksdb

Memory leak reported out of __cxa_thread_atexit

Open
#5,931 12 comments 3 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
32.1k
Forks
6.9k
Avg merge
32m
Merged PRs (30d)
1

Description

We upgraded to 6.2.4 recently and have started seeing leaks reported out of __cxa_thread_atexit on linux. These leaks are reproducible with db_bench.

```
==1== 24 bytes in 1 blocks are definitely lost in loss record 536 of 1,353
==1== at 0x4C29680: operator new(unsigned long, std::nothrow_t const&) (vg_replace_malloc.c:385)
==1== by 0x7A5115: __cxa_thread_atexit (in /rocksdb-host/db_bench)
==1== by 0x5FD8E8: UnknownInlinedFun (instrumented_mutex.cc:69)
==1== by 0x5FD8E8: rocksdb::InstrumentedMutex::Lock() (instrumented_mutex.cc:24)
==1== by 0x4EBA45: InstrumentedMutexLock (instrumented_mutex.h:56)
==1== by 0x4EBA45: rocksdb::DBImpl::BackgroundCallFlush(rocksdb::Env::Priority) (db_impl_compaction_flush.cc:2154)
==1== by 0x6BBF6A: operator() (std_function.h:706)
==1== by 0x6BBF6A: rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) (threadpool_imp.cc:266)
==1== by 0x6BC0E4: rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) (threadpool_imp.cc:307)
==1== by 0x7A521E: execute_native_thread_routine (in /rocksdb-host/db_bench)
==1== by 0x4E3AAA0: start_thread (in /lib64/libpthread-2.12.so)
==1== by 0x67D7C4C: clone (in /lib64/libc-2.12.so)
==1==
==1== 24 bytes in 1 blocks are definitely lost in loss record 537 of 1,353
==1== at 0x4C29680: operator new(unsigned long, std::nothrow_t const&) (vg_replace_malloc.c:385)
==1== by 0x7A5115: __cxa_thread_atexit (in /rocksdb-host/db_bench)
==1== by 0x5FD8E8: UnknownInlinedFun (instrumented_mutex.cc:69)
==1== by 0x5FD8E8: rocksdb::InstrumentedMutex::Lock() (instrumented_mutex.cc:24)
==1== by 0x4EC3B8: InstrumentedMutexLock (instrumented_mutex.h:56)
==1== by 0x4EC3B8: rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) (db_impl_compaction_flush.cc:2231)
==1== by 0x4EC8D1: rocksdb::DBImpl::BGWorkCompaction(void*) (db_impl_compaction_flush.cc:2022)
==1== by 0x6BBF6A: operator() (std_function.h:706)
==1== by 0x6BBF6A: rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) (threadpool_imp.cc:266)
==1== by 0x6BC0E4: rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) (threadpool_imp.cc:307)
==1== by 0x7A521E: execute_native_thread_routine (in /rocksdb-host/db_bench)
==1== by 0x4E3AAA0: start_thread (in /lib64/libpthread-2.12.so)
==1== by 0x67D7C4C: clone (in /lib64/libc-2.12.so)
```

### Expected behavior
No leaks reported by Valgrind or AddressSanitizer

### Actual behavior
Leaks reported by Valgrind and AddressSanitizer

### Steps to reproduce the behavior:
[Dockerfile to build rocksdb and valgrind](https://github.com/facebook/rocksdb/files/3734788/Dockerfile.txt)

db_bench command:
```
docker run --rm rocks-leak valgrind --leak-check=full ./db_bench --benchmarks="fillrandom,readrandom" --num 10000000
```

leaks reported:
```
==1==
==1== HEAP SUMMARY:
==1== in use at exit: 8,615,691 bytes in 7,555 blocks
==1== total heap usage: 98,705,609 allocs, 98,698,054 frees, 161,920,894,533 bytes allocated
==1==
==1== 24 bytes in 1 blocks are definitely lost in loss record 536 of 1,353
==1== at 0x4C29680: operator new(unsigned long, std::nothrow_t const&) (vg_replace_malloc.c:385)
==1== by 0x7A5115: __cxa_thread_atexit (in /rocksdb-host/db_bench)
==1== by 0x5FD8E8: UnknownInlinedFun (instrumented_mutex.cc:69)
==1== by 0x5FD8E8: rocksdb::InstrumentedMutex::Lock() (instrumented_mutex.cc:24)
==1== by 0x4EBA45: InstrumentedMutexLock (instrumented_mutex.h:56)
==1== by 0x4EBA45: rocksdb::DBImpl::BackgroundCallFlush(rocksdb::Env::Priority) (db_impl_compaction_flush.cc:2154)
==1== by 0x6BBF6A: operator() (std_function.h:706)
==1== by 0x6BBF6A: rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) (threadpool_imp.cc:266)
==1== by 0x6BC0E4: rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) (threadpool_imp.cc:307)
==1== by 0x7A521E: execute_native_thread_routine (in /rocksdb-host/db_bench)
==1== by 0x4E3AAA0: start_thread (in /lib64/libpthread-2.12.so)
==1== by 0x67D7C4C: clone (in /lib64/libc-2.12.so)
==1==
==1== 24 bytes in 1 blocks are definitely lost in loss record 537 of 1,353
==1== at 0x4C29680: operator new(unsigned long, std::nothrow_t const&) (vg_replace_malloc.c:385)
==1== by 0x7A5115: __cxa_thread_atexit (in /rocksdb-host/db_bench)
==1== by 0x5FD8E8: UnknownInlinedFun (instrumented_mutex.cc:69)
==1== by 0x5FD8E8: rocksdb::InstrumentedMutex::Lock() (instrumented_mutex.cc:24)
==1== by 0x4EC3B8: InstrumentedMutexLock (instrumented_mutex.h:56)
==1== by 0x4EC3B8: rocksdb::DBImpl::BackgroundCallCompaction(rocksdb::DBImpl::PrepickedCompaction*, rocksdb::Env::Priority) (db_impl_compaction_flush.cc:2231)
==1== by 0x4EC8D1: rocksdb::DBImpl::BGWorkCompaction(void*) (db_impl_compaction_flush.cc:2022)
==1== by 0x6BBF6A: operator() (std_function.h:706)
==1== by 0x6BBF6A: rocksdb::ThreadPoolImpl::Impl::BGThread(unsigned long) (threadpool_imp.cc:266)
==1== by 0x6BC0E4: rocksdb::ThreadPoolImpl::Impl::BGThreadWrapper(void*) (threadpool_imp.cc:307)
==1== by 0x7A521E: execute_native_thread_routine (in /rocksdb-host/db_bench)
==1== by 0x4E3AAA0: start_thread (in /lib64/libpthread-2.12.so)
==1== by 0x67D7C4C: clone (in /lib64/libc-2.12.so)
==1==
==1== 11,996 bytes in 3 blocks are possibly lost in loss record 1,347 of 1,353
==1== at 0x4C29F64: operator new[](unsigned long) (vg_replace_malloc.c:431)
==1== by 0x66D317: AllocateBlock (memory_allocator.h:35)
==1== by 0x66D317: rocksdb::UncompressBlockContentsForCompressionType(rocksdb::UncompressionInfo const&, char const*, unsigned long, rocksdb::BlockContents*, unsigned int, rocksdb::ImmutableCFOptions const&, rocksdb::MemoryAllocator*) (format.cc:300)
==1== by 0x66DB86: rocksdb::UncompressBlockContents(rocksdb::UncompressionInfo const&, char const*, unsigned long, rocksdb::BlockContents*, unsigned int, rocksdb::ImmutableCFOptions const&, rocksdb::MemoryAllocator*) (format.cc:409)
==1== by 0x660341: rocksdb::BlockFetcher::ReadBlockContents() (block_fetcher.cc:252)
==1== by 0x64EB4A: rocksdb::BlockBasedTable::MaybeReadBlockAndLoadToCache(rocksdb::FilePrefetchBuffer*, rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::UncompressionDict const&, rocksdb::BlockBasedTable::CachableEntry*, bool, rocksdb::GetContext*) (block_based_table_reader.cc:2140)
==1== by 0x65B9FE: rocksdb::DataBlockIter* rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::DataBlockIter*, bool, bool, bool, rocksdb::GetContext*, rocksdb::Status, rocksdb::FilePrefetchBuffer*) (block_based_table_reader.cc:1983)
==1== by 0x65744B: rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::SliceTransform const*, bool) (block_based_table_reader.cc:2765)
==1== by 0x583CF4: rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::SliceTransform const*, rocksdb::HistogramImpl*, bool, int) (table_cache.cc:392)
==1== by 0x5A4589: rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, unsigned long*, bool*, bool*, unsigned long*, rocksdb::ReadCallback*, bool*) (version_set.cc:1636)
==1== by 0x4BB18D: rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*, rocksdb::ReadCallback*, bool*) (db_impl.cc:1447)
==1== by 0x4BB3C6: rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*) (db_impl.cc:1356)
==1== by 0x44FDD2: rocksdb::Benchmark::ReadRandom(rocksdb::ThreadState*) (db_bench_tool.cc:4597)
==1==
==1== 8,553,231 (87 direct, 8,553,144 indirect) bytes in 1 blocks are definitely lost in loss record 1,353 of 1,353
==1== at 0x4C29F64: operator new[](unsigned long) (vg_replace_malloc.c:431)
==1== by 0x4669AD: rocksdb::LRUCacheShard::Insert(rocksdb::Slice const&, unsigned int, void*, unsigned long, void (*)(rocksdb::Slice const&, void*), rocksdb::Cache::Handle**, rocksdb::Cache::Priority) (lru_cache.cc:348)
==1== by 0x467701: rocksdb::ShardedCache::Insert(rocksdb::Slice const&, void*, unsigned long, void (*)(rocksdb::Slice const&, void*), rocksdb::Cache::Handle**, rocksdb::Cache::Priority) (sharded_cache.cc:55)
==1== by 0x64A91F: rocksdb::BlockBasedTable::PutDataBlockToCache(rocksdb::Slice const&, rocksdb::Slice const&, rocksdb::Cache*, rocksdb::Cache*, rocksdb::ReadOptions const&, rocksdb::ImmutableCFOptions const&, rocksdb::BlockBasedTable::CachableEntry*, rocksdb::BlockContents*, rocksdb::CompressionType, unsigned int, rocksdb::UncompressionDict const&, unsigned long, unsigned long, rocksdb::MemoryAllocator*, bool, rocksdb::Cache::Priority, rocksdb::GetContext*) (block_based_table_reader.cc:1558)
==1== by 0x64EC1E: rocksdb::BlockBasedTable::MaybeReadBlockAndLoadToCache(rocksdb::FilePrefetchBuffer*, rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::UncompressionDict const&, rocksdb::BlockBasedTable::CachableEntry*, bool, rocksdb::GetContext*) (block_based_table_reader.cc:2148)
==1== by 0x65B9FE: rocksdb::DataBlockIter* rocksdb::BlockBasedTable::NewDataBlockIterator(rocksdb::BlockBasedTable::Rep*, rocksdb::ReadOptions const&, rocksdb::BlockHandle const&, rocksdb::DataBlockIter*, bool, bool, bool, rocksdb::GetContext*, rocksdb::Status, rocksdb::FilePrefetchBuffer*) (block_based_table_reader.cc:1983)
==1== by 0x65744B: rocksdb::BlockBasedTable::Get(rocksdb::ReadOptions const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::SliceTransform const*, bool) (block_based_table_reader.cc:2765)
==1== by 0x583CF4: rocksdb::TableCache::Get(rocksdb::ReadOptions const&, rocksdb::InternalKeyComparator const&, rocksdb::FileMetaData const&, rocksdb::Slice const&, rocksdb::GetContext*, rocksdb::SliceTransform const*, rocksdb::HistogramImpl*, bool, int) (table_cache.cc:392)
==1== by 0x5A4589: rocksdb::Version::Get(rocksdb::ReadOptions const&, rocksdb::LookupKey const&, rocksdb::PinnableSlice*, rocksdb::Status*, rocksdb::MergeContext*, unsigned long*, bool*, bool*, unsigned long*, rocksdb::ReadCallback*, bool*) (version_set.cc:1636)
==1== by 0x4BB18D: rocksdb::DBImpl::GetImpl(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*, bool*, rocksdb::ReadCallback*, bool*) (db_impl.cc:1447)
==1== by 0x4BB3C6: rocksdb::DBImpl::Get(rocksdb::ReadOptions const&, rocksdb::ColumnFamilyHandle*, rocksdb::Slice const&, rocksdb::PinnableSlice*) (db_impl.cc:1356)
==1== by 0x44FDD2: rocksdb::Benchmark::ReadRandom(rocksdb::ThreadState*) (db_bench_tool.cc:4597)
==1==
==1== LEAK SUMMARY:
==1== definitely lost: 135 bytes in 3 blocks
==1== indirectly lost: 8,553,144 bytes in 6,205 blocks
==1== possibly lost: 11,996 bytes in 3 blocks
==1== still reachable: 50,416 bytes in 1,344 blocks
==1== of which reachable via heuristic:
==1== stdstring : 864 bytes in 9 blocks
==1== suppressed: 0 bytes in 0 blocks
==1== Reachable blocks (those to which a pointer was found) are not shown.
==1== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==1==
==1== For lists of detected and suppressed errors, rerun with: -s
==1== ERROR SUMMARY: 4 errors from 4 contexts (suppressed: 4 from 4)
```

[full output](https://github.com/facebook/rocksdb/files/3734774/output.txt)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.