StarRocks / StarRocks/starrocks

[Bug] SIGSEGV in `ColumnReader::~ColumnReader()` calling `protobuf::Message::SpaceUsedLong()` during `publish_version` / lake tablet metadata caching (shared-data mode)

Open
#71,283 7 comments 0 reactions 2 assignees View on GitHub

@kevincai is already working on this.

Since Apr 8, 2026.

type/bug
Dominant language
Java
Stars
12.1k
Forks
2.6k
Avg merge
1d 12h
Merged PRs (30d)
663

Description

[Bug] SIGSEGV in ColumnReader::~ColumnReader() calling protobuf::Message::SpaceUsedLong() during publish_version / lake tablet metadata caching (shared-data mode)

Environment

  • StarRocks version: 4.0.8 RELEASE (build 2090bfc distro ubuntu arch x86_64)
  • Deployment: Kubernetes, CN-only cluster (shared-data / lake mode), 15 CN pods
  • Component: CN (Compute Node)
  • Run mode: shared_data

Steps to reproduce the behavior (Required)

The crash occurs during background publish_version operations — no explicit user query is needed to trigger it. The conditions observed:

  1. Run a CN cluster in shared_data mode with datacache enabled.
  2. Sustain continuous data ingestion via Routine Load (Kafka) producing frequent version publishes.
  3. Observe SIGSEGV crash during compaction tasks.

Expected behavior (Required)

CN pod runs stably. Tablet metadata caching and version publishing complete without crashing.


Real behavior (Required)

CN pod crashes with SIGSEGV. 5 out of 15 CN pods crashed within a ~30-minute window (14:59–15:28 UTC+2). Two variants observed depending on which protobuf method is hit at the crash site.

Stack trace — Variant A: Message::SpaceUsedLong (CN-3, CN-14)
*** Aborted at 1775221180 (unix time) try "date -d @1775221180" if you are using GNU date ***
PC: @         0x1251cf83 google::protobuf::Message::SpaceUsedLong() const
*** SIGSEGV (@0x98) received by PID 22 (TID 0x7f5e1d9fe640) LWP(806) from PID 152; stack trace: ***
    @     0x7f5f0ee7dee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11fc49a8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f5f0ee26520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @         0x1251cf83 google::protobuf::Message::SpaceUsedLong() const
    @          0xd72453d starrocks::ColumnReader::~ColumnReader()
    @          0xd712479 std::_Rb_tree<int, std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >*)
    @          0xd716008 starrocks::Segment::~Segment()
    @          0x8c99206 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()
    @          0xdd2766c starrocks::lake::Metacache::cache_value_deleter(starrocks::CacheKey const&, void*)
    @          0xe45ba5d starrocks::LRUCache::insert(starrocks::CacheKey const&, unsigned int, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
    @          0xe45bd66 starrocks::ShardedLRUCache::insert(starrocks::CacheKey const&, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
    @          0xdd2492e starrocks::lake::Metacache::cache_tablet_metadata(std::basic_string_view<char, std::char_traits<char> >, std::shared_ptr<starrocks::TabletMetadataPB const>)
    @          0xdc5b899 starrocks::lake::TabletManager::cache_tablet_metadata(std::shared_ptr<starrocks::TabletMetadataPB const> const&)
    @          0xe265313 starrocks::lake::NonPrimaryKeyTxnLogApplier::finish()
    @          0xe25fe3a starrocks::lake::publish_version(starrocks::lake::TabletManager*, long, long, long, std::span<starrocks::TxnInfoPB const, 18446744073709551615ul>, bool)
    @          0xe23ea1b starrocks::LakeServiceImpl::publish_version(google::protobuf::RpcController*, starrocks::PublishVersionRequest const*, starrocks::PublishVersionResponse*, google::protobuf::Closure*)::{lambda()#1}::operator()() const
    @          0xe439e68 std::_Function_handler<void (), starrocks::ConcurrencyLimitedThreadPoolToken::submit(std::shared_ptr<starrocks::Runnable>, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
    @          0xe44172e starrocks::ThreadPool::dispatch_thread()
    @          0xe4381a5 starrocks::Thread::supervise_thread(void*)
    @     0x7f5f0ee78ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f5f0ef0a8d0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268cf)

Same stack also observed on CN-14 at 1775222880 (same PC 0x1251cf83, same call chain).

Stack trace — Variant B: Reflection::SpaceUsedLong (CN-10)
*** Aborted at 1775221754 (unix time) try "date -d @1775221754" if you are using GNU date ***
PC: @         0x12502dd9 google::protobuf::Reflection::SpaceUsedLong(google::protobuf::Message const&) const
*** SIGSEGV (@0x88) received by PID 22 (TID 0x7f863daf1640) LWP(806) from PID 136; stack trace: ***
    @     0x7f873cf17ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11fc49a8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f873cec0520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @         0x12502dd9 google::protobuf::Reflection::SpaceUsedLong(google::protobuf::Message const&) const
    @          0xd724685 starrocks::ColumnReader::~ColumnReader()
    @          0xd712479 std::_Rb_tree<int, std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >*)
    @          0xd71225c std::_Rb_tree<int, std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > > >::erase(std::_Rb_tree_const_iterator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >)
    @          0xd716008 starrocks::Segment::~Segment()
    @          0x8c99206 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()
    @          0xdd2766c starrocks::lake::Metacache::cache_value_deleter(starrocks::CacheKey const&, void*)
    @          0xe45ba5d starrocks::LRUCache::insert(starrocks::CacheKey const&, unsigned int, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
    @          0xe45bd66 starrocks::ShardedLRUCache::insert(starrocks::CacheKey const&, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
    @          0xdd2492e starrocks::lake::Metacache::cache_tablet_metadata(std::basic_string_view<char, std::char_traits<char> >, std::shared_ptr<starrocks::TabletMetadataPB const>)
    @          0xdc5b899 starrocks::lake::TabletManager::cache_tablet_metadata(std::shared_ptr<starrocks::TabletMetadataPB const> const&)
    @          0xe265313 starrocks::lake::NonPrimaryKeyTxnLogApplier::finish()
    @          0xe25fe3a starrocks::lake::publish_version(starrocks::lake::TabletManager*, long, long, long, std::span<starrocks::TxnInfoPB const, 18446744073709551615ul>, bool)
    @          0xe23ea1b starrocks::LakeServiceImpl::publish_version(google::protobuf::RpcController*, starrocks::PublishVersionRequest const*, starrocks::PublishVersionResponse*, google::protobuf::Closure*)::{lambda()#1}::operator()() const
    @          0xe439e68 std::_Function_handler<void (), starrocks::ConcurrencyLimitedThreadPoolToken::submit(std::shared_ptr<starrocks::Runnable>, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
    @          0xe44172e starrocks::ThreadPool::dispatch_thread()
    @          0xe4381a5 starrocks::Thread::supervise_thread(void*)
    @     0x7f873cf12ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f873cfa48d0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268cf)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.