apache / apache/doris

BE SIGSEGV in VStatisticsIterator::init() / Segment::new_column_iterator() during segment finalize for VARIANT column

Open
#66,911 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

## Version
Apache Doris **4.0.5-rc01**

## What happened
During a sustained bulk-load workload (`INSERT INTO ... FROM LOCAL(...)` loading Parquet files via the local-disk TVF path, ~20 files / batch, `parallel_pipeline_task_num=20`), the BE process crashed with `SIGSEGV` twice over a ~2 hour window, both times with the same signature. The crash occurs on the **write path**, not a query/scan path: the crashing thread's preceding log lines show it actively flushing and finalizing freshly-written segments (`segment_creator.cpp:271] tablet_id:..., flushing rowset_dir:...`) for the table being bulk-inserted into, immediately before the fault.

The table being written to has one `VARIANT`-typed column among a mix of standard-typed columns, and 3 inverted indexes (on other, non-variant columns). Process memory was healthy at crash time (~49-50 GB used, well under both the 216 GB soft limit and 240 GB hard limit) — this does **not** look like a memory-pressure crash.

## Backtrace (top frames)

```
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at .../common/signal_handler.h:420
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in libjvm.so
2# JVM_handle_linux_signal in libjvm.so
3# 0x00007FDDF0128520 in libc.so.6
4# doris::segment_v2::Segment::new_column_iterator(doris::TabletColumn const&, std::unique_ptr*, doris::StorageReadOptions const*, std::unordered_map>>> const*) at .../olap/rowset/segment_v2/segment.cpp:683
5# doris::vectorized::VStatisticsIterator::init(doris::StorageReadOptions const&) at .../vec/olap/vgeneric_iterators.cpp:54
6# doris::segment_v2::Segment::new_iterator(...) in /opt/apache-doris/be/lib/doris_be
7# doris::segment_v2::LazyInitSegmentIterator::init(doris::StorageReadOptions const&) at .../olap/rowset/segment_v2/lazy_init_segment_iterator.cpp:48
```

## What appears to trigger it

Immediately preceding the crash, the same thread ID logged repeated `segment_creator.cpp:271] tablet_id:, flushing rowset_dir: ..., rowset_id:...` lines for several different tablets in quick succession (bulk-load segment finalization), interleaved with `variant_statistics.h:37] num subcolumns 0, num sparse columns 0` lines. The crash happens in the call chain that builds a column iterator to compute embedded zone-map/min-max statistics for a just-flushed segment, specifically going through the `SparseColumnCache`-aware overload of `new_column_iterator`, which strongly suggests the fault is in statistics construction for the `VARIANT` column's sparse-subcolumn handling at segment-finalize time — not in the write of the segment data itself (which completed and closed cleanly per the preceding `load_stream_writer.cpp:174] file ... closed` lines).

This crashed **twice** in one day under the same general workload (bulk INSERT into this table), both times during segment finalization, both times with process memory well within limits.

## Environment
- Doris 4.0.5-rc01, 4-BE cluster (3 FEs), `replication_allocation=1` on the affected table
- Table: `DUPLICATE KEY` model, one `VARIANT` column among standard-typed columns, 3 inverted indexes
- Workload: bulk `INSERT INTO ... FROM LOCAL(...)` (local-disk Parquet TVF), `parallel_pipeline_task_num=20`, batches of ~20 files (~6-8M rows) per statement

Contributor guide

Open the contributing guide

Research direction

Start by tracing the call chain from segment_creator.cpp and variant_statistics.h through vec/olap/vgeneric_iterators.cpp:54 and olap/rowset/segment_v2/segment.cpp:683. Reproduce the bulk INSERT workload with a VARIANT column, then inspect the SparseColumnCache-aware iterator path and the surrounding finalize logs. Done means the segment-finalization workload no longer produces the reported SIGSEGV.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.