BE SIGSEGV in VStatisticsIterator::init() / Segment::new_column_iterator() during segment finalize for VARIANT column
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
## Version
Apache Doris **4.0.5-rc01**
## What happened
During a sustained bulk-load workload (`INSERT INTO ... FROM LOCAL(...)` loading Parquet files via the local-disk TVF path, ~20 files / batch, `parallel_pipeline_task_num=20`), the BE process crashed with `SIGSEGV` twice over a ~2 hour window, both times with the same signature. The crash occurs on the **write path**, not a query/scan path: the crashing thread's preceding log lines show it actively flushing and finalizing freshly-written segments (`segment_creator.cpp:271] tablet_id:..., flushing rowset_dir:...`) for the table being bulk-inserted into, immediately before the fault.
The table being written to has one `VARIANT`-typed column among a mix of standard-typed columns, and 3 inverted indexes (on other, non-variant columns). Process memory was healthy at crash time (~49-50 GB used, well under both the 216 GB soft limit and 240 GB hard limit) — this does **not** look like a memory-pressure crash.
## Backtrace (top frames)
```
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at .../common/signal_handler.h:420
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in libjvm.so
2# JVM_handle_linux_signal in libjvm.so
3# 0x00007FDDF0128520 in libc.so.6
4# doris::segment_v2::Segment::new_column_iterator(doris::TabletColumn const&, std::unique_ptr*, doris::StorageReadOptions const*, std::unordered_map>>> const*) at .../olap/rowset/segment_v2/segment.cpp:683
5# doris::vectorized::VStatisticsIterator::init(doris::StorageReadOptions const&) at .../vec/olap/vgeneric_iterators.cpp:54
6# doris::segment_v2::Segment::new_iterator(...) in /opt/apache-doris/be/lib/doris_be
7# doris::segment_v2::LazyInitSegmentIterator::init(doris::StorageReadOptions const&) at .../olap/rowset/segment_v2/lazy_init_segment_iterator.cpp:48
```
## What appears to trigger it
Immediately preceding the crash, the same thread ID logged repeated `segment_creator.cpp:271] tablet_id:, flushing rowset_dir: ..., rowset_id:...` lines for several different tablets in quick succession (bulk-load segment finalization), interleaved with `variant_statistics.h:37] num subcolumns 0, num sparse columns 0` lines. The crash happens in the call chain that builds a column iterator to compute embedded zone-map/min-max statistics for a just-flushed segment, specifically going through the `SparseColumnCache`-aware overload of `new_column_iterator`, which strongly suggests the fault is in statistics construction for the `VARIANT` column's sparse-subcolumn handling at segment-finalize time — not in the write of the segment data itself (which completed and closed cleanly per the preceding `load_stream_writer.cpp:174] file ... closed` lines).
This crashed **twice** in one day under the same general workload (bulk INSERT into this table), both times during segment finalization, both times with process memory well within limits.
## Environment
- Doris 4.0.5-rc01, 4-BE cluster (3 FEs), `replication_allocation=1` on the affected table
- Table: `DUPLICATE KEY` model, one `VARIANT` column among standard-typed columns, 3 inverted indexes
- Workload: bulk `INSERT INTO ... FROM LOCAL(...)` (local-disk Parquet TVF), `parallel_pipeline_task_num=20`, batches of ~20 files (~6-8M rows) per statement
Contributor guide
Research direction
Start by tracing the call chain from segment_creator.cpp and variant_statistics.h through vec/olap/vgeneric_iterators.cpp:54 and olap/rowset/segment_v2/segment.cpp:683. Reproduce the bulk INSERT workload with a VARIANT column, then inspect the SparseColumnCache-aware iterator path and the surrounding finalize logs. Done means the segment-finalization workload no longer produces the reported SIGSEGV.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100