StarRocks / StarRocks/starrocks
[Bug] SIGSEGV in `ColumnReader::~ColumnReader()` calling `protobuf::Message::SpaceUsedLong()` during `publish_version` / lake tablet metadata caching (shared-data mode)
Open
@kevincai is already working on this.
Since Apr 8, 2026.
type/bug
- Dominant language
- Java
- Stars
- 12.1k
- Forks
- 2.6k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 663
Description
[Bug] SIGSEGV in ColumnReader::~ColumnReader() calling protobuf::Message::SpaceUsedLong() during publish_version / lake tablet metadata caching (shared-data mode)
Environment
- StarRocks version:
4.0.8 RELEASE (build 2090bfc distro ubuntu arch x86_64) - Deployment: Kubernetes, CN-only cluster (shared-data / lake mode), 15 CN pods
- Component: CN (Compute Node)
- Run mode:
shared_data
Steps to reproduce the behavior (Required)
The crash occurs during background publish_version operations — no explicit user query is needed to trigger it. The conditions observed:
- Run a CN cluster in
shared_datamode with datacache enabled. - Sustain continuous data ingestion via Routine Load (Kafka) producing frequent version publishes.
- Observe SIGSEGV crash during compaction tasks.
Expected behavior (Required)
CN pod runs stably. Tablet metadata caching and version publishing complete without crashing.
Real behavior (Required)
CN pod crashes with SIGSEGV. 5 out of 15 CN pods crashed within a ~30-minute window (14:59–15:28 UTC+2). Two variants observed depending on which protobuf method is hit at the crash site.
Stack trace — Variant A: Message::SpaceUsedLong (CN-3, CN-14)
*** Aborted at 1775221180 (unix time) try "date -d @1775221180" if you are using GNU date ***
PC: @ 0x1251cf83 google::protobuf::Message::SpaceUsedLong() const
*** SIGSEGV (@0x98) received by PID 22 (TID 0x7f5e1d9fe640) LWP(806) from PID 152; stack trace: ***
@ 0x7f5f0ee7dee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
@ 0x11fc49a8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
@ 0x7f5f0ee26520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
@ 0x1251cf83 google::protobuf::Message::SpaceUsedLong() const
@ 0xd72453d starrocks::ColumnReader::~ColumnReader()
@ 0xd712479 std::_Rb_tree<int, std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >*)
@ 0xd716008 starrocks::Segment::~Segment()
@ 0x8c99206 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()
@ 0xdd2766c starrocks::lake::Metacache::cache_value_deleter(starrocks::CacheKey const&, void*)
@ 0xe45ba5d starrocks::LRUCache::insert(starrocks::CacheKey const&, unsigned int, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
@ 0xe45bd66 starrocks::ShardedLRUCache::insert(starrocks::CacheKey const&, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
@ 0xdd2492e starrocks::lake::Metacache::cache_tablet_metadata(std::basic_string_view<char, std::char_traits<char> >, std::shared_ptr<starrocks::TabletMetadataPB const>)
@ 0xdc5b899 starrocks::lake::TabletManager::cache_tablet_metadata(std::shared_ptr<starrocks::TabletMetadataPB const> const&)
@ 0xe265313 starrocks::lake::NonPrimaryKeyTxnLogApplier::finish()
@ 0xe25fe3a starrocks::lake::publish_version(starrocks::lake::TabletManager*, long, long, long, std::span<starrocks::TxnInfoPB const, 18446744073709551615ul>, bool)
@ 0xe23ea1b starrocks::LakeServiceImpl::publish_version(google::protobuf::RpcController*, starrocks::PublishVersionRequest const*, starrocks::PublishVersionResponse*, google::protobuf::Closure*)::{lambda()#1}::operator()() const
@ 0xe439e68 std::_Function_handler<void (), starrocks::ConcurrencyLimitedThreadPoolToken::submit(std::shared_ptr<starrocks::Runnable>, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
@ 0xe44172e starrocks::ThreadPool::dispatch_thread()
@ 0xe4381a5 starrocks::Thread::supervise_thread(void*)
@ 0x7f5f0ee78ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f5f0ef0a8d0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268cf)
Same stack also observed on CN-14 at 1775222880 (same PC 0x1251cf83, same call chain).
Stack trace — Variant B: Reflection::SpaceUsedLong (CN-10)
*** Aborted at 1775221754 (unix time) try "date -d @1775221754" if you are using GNU date ***
PC: @ 0x12502dd9 google::protobuf::Reflection::SpaceUsedLong(google::protobuf::Message const&) const
*** SIGSEGV (@0x88) received by PID 22 (TID 0x7f863daf1640) LWP(806) from PID 136; stack trace: ***
@ 0x7f873cf17ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
@ 0x11fc49a8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
@ 0x7f873cec0520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
@ 0x12502dd9 google::protobuf::Reflection::SpaceUsedLong(google::protobuf::Message const&) const
@ 0xd724685 starrocks::ColumnReader::~ColumnReader()
@ 0xd712479 std::_Rb_tree<int, std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > > >::_M_erase(std::_Rb_tree_node<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >*)
@ 0xd71225c std::_Rb_tree<int, std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > >, std::_Select1st<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > > >::erase(std::_Rb_tree_const_iterator<std::pair<int const, std::unique_ptr<starrocks::ColumnReader, std::default_delete<starrocks::ColumnReader> > > >)
@ 0xd716008 starrocks::Segment::~Segment()
@ 0x8c99206 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release_last_use_cold()
@ 0xdd2766c starrocks::lake::Metacache::cache_value_deleter(starrocks::CacheKey const&, void*)
@ 0xe45ba5d starrocks::LRUCache::insert(starrocks::CacheKey const&, unsigned int, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
@ 0xe45bd66 starrocks::ShardedLRUCache::insert(starrocks::CacheKey const&, void*, unsigned long, void (*)(starrocks::CacheKey const&, void*), starrocks::CachePriority)
@ 0xdd2492e starrocks::lake::Metacache::cache_tablet_metadata(std::basic_string_view<char, std::char_traits<char> >, std::shared_ptr<starrocks::TabletMetadataPB const>)
@ 0xdc5b899 starrocks::lake::TabletManager::cache_tablet_metadata(std::shared_ptr<starrocks::TabletMetadataPB const> const&)
@ 0xe265313 starrocks::lake::NonPrimaryKeyTxnLogApplier::finish()
@ 0xe25fe3a starrocks::lake::publish_version(starrocks::lake::TabletManager*, long, long, long, std::span<starrocks::TxnInfoPB const, 18446744073709551615ul>, bool)
@ 0xe23ea1b starrocks::LakeServiceImpl::publish_version(google::protobuf::RpcController*, starrocks::PublishVersionRequest const*, starrocks::PublishVersionResponse*, google::protobuf::Closure*)::{lambda()#1}::operator()() const
@ 0xe439e68 std::_Function_handler<void (), starrocks::ConcurrencyLimitedThreadPoolToken::submit(std::shared_ptr<starrocks::Runnable>, std::chrono::time_point<std::chrono::_V2::system_clock, std::chrono::duration<long, std::ratio<1l, 1000000000l> > >)::{lambda()#1}>::_M_invoke(std::_Any_data const&)
@ 0xe44172e starrocks::ThreadPool::dispatch_thread()
@ 0xe4381a5 starrocks::Thread::supervise_thread(void*)
@ 0x7f873cf12ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
@ 0x7f873cfa48d0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268cf)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.