apache / apache/doris

[Bug] BE segfault in Arrow Flight SQL DoGet when using public_host (ILB proxy)

Open
#62,217 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

# BE Segfault in Arrow Flight SQL DoGet When Using `public_host` (ILB Proxy)

## Environment

- **Doris Version:** 4.0.3
- **Deployment Mode:** Disaggregated (compute-storage decoupled)
- **Platform:** GKE

## Setup

We access a prod Doris cluster from a separate stage cluster via Arrow Flight SQL. Since the stage cluster cannot resolve prod's internal K8s DNS, we use L4 Internal Load Balancers (ILBs) with the following BE config:

```
public_host = 10.142.0.127
arrow_flight_sql_proxy_port = 8050
```

The BE ILB (`10.142.0.127:8050`) routes to 3 BE pods in a dedicated compute group (`stage_to_prod_read`). FE returns the ILB IP as the endpoint in FlightInfo, so the client always connects to the ILB, which routes to a random BE.

## Issue

All 3 BE pods in this compute group frequently segfault in the Arrow Flight SQL `DoGet` handler. Each pod has 20+ restarts over 3 days.

We have another compute group (`risk_minutely`) that also uses Arrow Flight SQL but with clients connecting **directly to BE pod FQDNs** (no ILB, no `public_host`). That compute group has **0 crashes**.

## Core Dump

Stack trace is identical across all pods and all crashes:

```
0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0]
2# JVM_handle_linux_signal
3# 0x00007B4F35EA2520 in libc.so.6
4# std::__cxx11::basic_string<...>::basic_string<...>(char const*, ...) at basic_string.h:707
5# arrow::flight::internal::TransportStatus::FromStatus(arrow::Status const&)
6# arrow::flight::transport::grpc::ToGrpcStatus(arrow::Status const&, grpc::ServerContext*)
7# GrpcServiceHandler::DoGet(grpc::ServerContext*, arrow::flight::protocol::Ticket const*, grpc::ServerWriter*)
8-17# grpc internals...

/opt/apache-doris/be/bin/start_be.sh: line 600: Segmentation fault (core dumped)
```

## Logs Immediately Before Crash

Every crash is preceded by `ArrowFlightBatchReader` packet sequence errors on port 8060 (brpc — BE-to-BE internal forwarding):

```
W20260408 05:12:51.681289 arrow_flight_batch_reader.cpp:54] ArrowFlightBatchReader fetch data receive packet failed, expect: 0, receive: 1, packet_seq=0, result=doris-cluster-stage-to-prod-read-0.doris-cluster-stage-to-prod-read.doris.svc.cluster.local:8060, finistId=24496d8a078d4ae1-b25284a3dc9fbfa8
```

## Client-Side Error

```
IO: [FlightSQL] error reading from server: EOF (Unavailable; DoGet: endpoint 1: [uri:"grpc+tcp://10.142.0.127:8050"])
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.