[Bug] BE segfault in Arrow Flight SQL DoGet when using public_host (ILB proxy)
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
# BE Segfault in Arrow Flight SQL DoGet When Using `public_host` (ILB Proxy)
## Environment
- **Doris Version:** 4.0.3
- **Deployment Mode:** Disaggregated (compute-storage decoupled)
- **Platform:** GKE
## Setup
We access a prod Doris cluster from a separate stage cluster via Arrow Flight SQL. Since the stage cluster cannot resolve prod's internal K8s DNS, we use L4 Internal Load Balancers (ILBs) with the following BE config:
```
public_host = 10.142.0.127
arrow_flight_sql_proxy_port = 8050
```
The BE ILB (`10.142.0.127:8050`) routes to 3 BE pods in a dedicated compute group (`stage_to_prod_read`). FE returns the ILB IP as the endpoint in FlightInfo, so the client always connects to the ILB, which routes to a random BE.
## Issue
All 3 BE pods in this compute group frequently segfault in the Arrow Flight SQL `DoGet` handler. Each pod has 20+ restarts over 3 days.
We have another compute group (`risk_minutely`) that also uses Arrow Flight SQL but with clients connecting **directly to BE pod FQDNs** (no ILB, no `public_host`). That compute group has **0 crashes**.
## Core Dump
Stack trace is identical across all pods and all crashes:
```
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
2# JVM_handle_linux_signal
3# 0x00007B4F35EA2520 in libc.so.6
4# std::__cxx11::basic_string<...>::basic_string<...>(char const*, ...) at basic_string.h:707
5# arrow::flight::internal::TransportStatus::FromStatus(arrow::Status const&)
6# arrow::flight::transport::grpc::ToGrpcStatus(arrow::Status const&, grpc::ServerContext*)
7# GrpcServiceHandler::DoGet(grpc::ServerContext*, arrow::flight::protocol::Ticket const*, grpc::ServerWriter*)
8-17# grpc internals...
/opt/apache-doris/be/bin/start_be.sh: line 600: Segmentation fault (core dumped)
```
## Logs Immediately Before Crash
Every crash is preceded by `ArrowFlightBatchReader` packet sequence errors on port 8060 (brpc — BE-to-BE internal forwarding):
```
W20260408 05:12:51.681289 arrow_flight_batch_reader.cpp:54] ArrowFlightBatchReader fetch data receive packet failed, expect: 0, receive: 1, packet_seq=0, result=doris-cluster-stage-to-prod-read-0.doris-cluster-stage-to-prod-read.doris.svc.cluster.local:8060, finistId=24496d8a078d4ae1-b25284a3dc9fbfa8
```
## Client-Side Error
```
IO: [FlightSQL] error reading from server: EOF (Unavailable; DoGet: endpoint 1: [uri:"grpc+tcp://10.142.0.127:8050"])
```
Contributor guide
Assessment
This issue has not been assessed yet.