SIGSEGV in BlockBasedTable DumpDataBlocks (Kafka Streams)
- Dominant language
- C++
- Stars
- 32.1k
- Forks
- 6.9k
- Avg merge
- 32m
- Merged PRs (30d)
- 1
Description
A project that I'm working on (Kafka Streams) uses RocksDB via JNI.
While running tests, I encountered a SIGSEGV.
Downstream issue: https://issues.apache.org/jira/browse/KAFKA-14555
### Expected behavior
No SIGSEGV
### Actual behavior
JVM Crash with the following stacktrace:
```
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00000001269f2f2c, pid=88913, tid=40199
#
# JRE version: OpenJDK Runtime Environment Corretto-17.0.4.9.1 (17.0.4.1+9) (build 17.0.4.1+9-LTS)
# Java VM: OpenJDK 64-Bit Server VM Corretto-17.0.4.9.1 (17.0.4.1+9-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, parallel gc, bsd-aarch64)
# Problematic frame:
# C [librocksdbjni15989196819046251041.jnilib+0x2def2c] _ZN7rocksdb15BlockBasedTable14DumpDataBlocksERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEE+0x1650
--------------- T H R E A D ---------------
Current thread is native
threadStack: [0x0000000171704000,0x0000000171787000], sp=0x0000000171784e90, free space=515k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [librocksdbjni15989196819046251041.jnilib+0x2def2c] _ZN7rocksdb15BlockBasedTable14DumpDataBlocksERNSt3__113basic_ostreamIcNS1_11char_traitsIcEEEE+0x1650
C [librocksdbjni15989196819046251041.jnilib+0x2cff98] _ZN7rocksdb15BlockBasedTable28PrefetchIndexAndFilterBlocksERKNS_11ReadOptionsEPNS_18FilePrefetchBufferEPNS_20InternalIteratorBaseINS_5SliceEEEPS0_bRKNS_22BlockBasedTableOptionsEimmPNS_23BlockCacheLookupContextE+0x354
C [librocksdbjni15989196819046251041.jnilib+0x2ce4d4] _ZN7rocksdb15BlockBasedTable4OpenERKNS_11ReadOptionsERKNS_16ImmutableOptionsERKNS_10EnvOptionsERKNS_22BlockBasedTableOptionsERKNS_21InternalKeyComparatorEONSt3__110unique_ptrINS_22RandomAccessFileReaderENSG_14default_deleteISI_EEEEyPNSH_INS_11TableReaderENSJ_ISN_EEEERKNSG_10shared_ptrIKNS_14SliceTransformEEEbbibybPNS_17TailPrefetchStatsEPNS_16BlockCacheTracerEmRKNSG_12basic_stringIcNSG_11char_traitsIcEENSG_9allocatorIcEEEEy+0xaa0
C [librocksdbjni15989196819046251041.jnilib+0x2bbd04] _ZNK7rocksdb22BlockBasedTableFactory14NewTableReaderERKNS_11ReadOptionsERKNS_18TableReaderOptionsEONSt3__110unique_ptrINS_22RandomAccessFileReaderENS7_14default_deleteIS9_EEEEyPNS8_INS_11TableReaderENSA_ISE_EEEEb+0x8c
C [librocksdbjni15989196819046251041.jnilib+0x18de18] _ZN7rocksdb10TableCache14GetTableReaderERKNS_11ReadOptionsERKNS_11FileOptionsERKNS_21InternalKeyComparatorERKNS_14FileDescriptorEbbPNS_13HistogramImplEPNSt3__110unique_ptrINS_11TableReaderENSF_14default_deleteISH_EEEERKNSF_10shared_ptrIKNS_14SliceTransformEEEbibmNS_11TemperatureE+0x418
C [librocksdbjni15989196819046251041.jnilib+0x18e5e8] _ZN7rocksdb10TableCache9FindTableERKNS_11ReadOptionsERKNS_11FileOptionsERKNS_21InternalKeyComparatorERKNS_14FileDescriptorEPPNS_5Cache6HandleERKNSt3__110shared_ptrIKNS_14SliceTransformEEEbbPNS_13HistogramImplEbibmNS_11TemperatureE+0x22c
C [librocksdbjni15989196819046251041.jnilib+0x18e96c] _ZN7rocksdb10TableCache11NewIteratorERKNS_11ReadOptionsERKNS_11FileOptionsERKNS_21InternalKeyComparatorERKNS_12FileMetaDataEPNS_18RangeDelAggregatorERKNSt3__110shared_ptrIKNS_14SliceTransformEEEPPNS_11TableReaderEPNS_13HistogramImplENS_17TableReaderCallerEPNS_5ArenaEbimPKNS_11InternalKeyESW_b+0x1ac
C [librocksdbjni15989196819046251041.jnilib+0x8fdc8] _ZN7rocksdb13CompactionJob25ProcessKeyValueCompactionEPNS0_18SubcompactionStateE+0x1be4
C [librocksdbjni15989196819046251041.jnilib+0x8d92c] _ZN7rocksdb13CompactionJob3RunEv+0xed8
C [librocksdbjni15989196819046251041.jnilib+0xfb318] _ZN7rocksdb6DBImpl20BackgroundCompactionEPbPNS_10JobContextEPNS_9LogBufferEPNS0_19PrepickedCompactionENS_3Env8PriorityE+0xbc8
C [librocksdbjni15989196819046251041.jnilib+0xf9484] _ZN7rocksdb6DBImpl24BackgroundCallCompactionEPNS0_19PrepickedCompactionENS_3Env8PriorityE+0xc0
C [librocksdbjni15989196819046251041.jnilib+0xf6f58] _ZN7rocksdb6DBImpl16BGWorkCompactionEPv+0x30
C [librocksdbjni15989196819046251041.jnilib+0x3561dc] _ZN7rocksdb14ThreadPoolImpl4Impl8BGThreadEm+0x1ec
C [librocksdbjni15989196819046251041.jnilib+0x35645c] _ZN7rocksdb14ThreadPoolImpl4Impl15BGThreadWrapperEPv+0x7c
C [librocksdbjni15989196819046251041.jnilib+0x357ed8] _ZN7rocksdb13NewThreadPoolEi+0x2b0
C [libsystem_pthread.dylib+0x726c] _pthread_start+0x94
```
Full log:
[hs_err_pid88913.log](https://github.com/facebook/rocksdb/files/10326091/hs_err_pid88913.log)
### Steps to reproduce the behavior
1. Clone Kafka https://github.com/apache/kafka from `trunk`
2. Run `./gradlew streams:cleanTest streams:test`
Note: I have only seen this failure once so far and have not yet verified these reproduction steps.
I am using macOS Monterey 12.6 with an arm64/aarch64 Apple Silicon M1 Max.
Contributor guide
Research direction
Begin by investigating the BlockBasedTable::DumpDataBlocks frame in the provided stack trace and try the Kafka Streams reproduction with ./gradlew streams:cleanTest streams:test on macOS arm64. The issue is resolved when the reproduction no longer causes a JVM SIGSEGV; the report notes that the steps have not been verified and the failure occurred only once.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100