[Bug] BE SIGSEGV in bvar::SeriesSampler::take_sample on Doris 4.1.3
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
Apache Doris 4.1.3
Environment:
Kubernetes
Compute-storage coupled mode
6 BE nodes
### What's Wrong?
Two different BE nodes crashed independently with the same SIGSEGV stack trace:
```
0# doris::signal::(anonymous namespace)::FailureSignalHandler(...)
1# PosixSignals::chained_handler(...)
2# JVM_handle_linux_signal
3# libc.so.6
4# bvar::Reducer,
bvar::detail::MinusFrom>::SeriesSampler::take_sample()
5# bvar::detail::SamplerCollector::run()
6# bvar::detail::SamplerCollector::sampling_thread(void*)
7# libc.so.6
8# libc.so.6
```
The same crash occurred independently on two different BEs.
This appears similar to #63193, which was addressed by #64040 and included in Doris 4.1.3.
There was no memory pressure at the time of the crashes:
BE RSS: ~113 GB / ~155 GB
BE process memory limit: 1080 GB
Physical memory: 1200 GB
### What You Expected?
BE should remain stable and should not terminate with SIGSEGV.
### How to Reproduce?
We do not currently have a deterministic standalone reproducer.
The crashes started while rewriting a large table with approximately 40 billion rows from storage format V2 to V3, while the cluster was also serving its normal production workload.
### Anything Else?
#64040 backports the AgentCombiner lifetime changes from apache/brpc#2949.
There is also a follow-up upstream fix, apache/brpc#3291, which addresses an additional heap-use-after-free involving ~AgentCombiner()/clear_all_agents() and TLS Agent destruction.
The same follow-up change does not appear to be present in the current Doris 4.1 brpc patch.
Reporting this because the bvar::SeriesSampler::take_sample() SIGSEGV is still reproducible on Doris 4.1.3 after the related #64040 fix.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
Start at the bvar::SeriesSampler::take_sample() stack trace and compare the Doris 4.1 brpc patch with the follow-up apache/brpc#3291 change. Done means identifying and addressing the remaining SIGSEGV cause, then validating BE stability under the table rewrite and production workload described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100