apache / apache/doris

[Bug] BE SIGSEGV in bvar::SeriesSampler::take_sample on Doris 4.1.3

Open
#66,895 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

Apache Doris 4.1.3
Environment:
Kubernetes
Compute-storage coupled mode
6 BE nodes

### What's Wrong?

Two different BE nodes crashed independently with the same SIGSEGV stack trace:
```
0# doris::signal::(anonymous namespace)::FailureSignalHandler(...)
1# PosixSignals::chained_handler(...)
2# JVM_handle_linux_signal
3# libc.so.6
4# bvar::Reducer,
bvar::detail::MinusFrom>::SeriesSampler::take_sample()
5# bvar::detail::SamplerCollector::run()
6# bvar::detail::SamplerCollector::sampling_thread(void*)
7# libc.so.6
8# libc.so.6
```
The same crash occurred independently on two different BEs.
This appears similar to #63193, which was addressed by #64040 and included in Doris 4.1.3.
There was no memory pressure at the time of the crashes:
BE RSS: ~113 GB / ~155 GB
BE process memory limit: 1080 GB
Physical memory: 1200 GB

### What You Expected?

BE should remain stable and should not terminate with SIGSEGV.

### How to Reproduce?

We do not currently have a deterministic standalone reproducer.
The crashes started while rewriting a large table with approximately 40 billion rows from storage format V2 to V3, while the cluster was also serving its normal production workload.

### Anything Else?

#64040 backports the AgentCombiner lifetime changes from apache/brpc#2949.
There is also a follow-up upstream fix, apache/brpc#3291, which addresses an additional heap-use-after-free involving ~AgentCombiner()/clear_all_agents() and TLS Agent destruction.
The same follow-up change does not appear to be present in the current Doris 4.1 brpc patch.
Reporting this because the bvar::SeriesSampler::take_sample() SIGSEGV is still reproducible on Doris 4.1.3 after the related #64040 fix.

### Are you willing to submit PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

Start at the bvar::SeriesSampler::take_sample() stack trace and compare the Doris 4.1 brpc patch with the follow-up apache/brpc#3291 change. Done means identifying and addressing the remaining SIGSEGV cause, then validating BE stability under the table rewrite and production workload described.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.