apache / apache/doris

[Bug] BE SIGSEGV and wrong result with nested lambda over array_agg result in array_map/array_count

Open
#63,824 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.9k
Forks
3.9k
Avg merge
2d 23h
Merged PRs (30d)
520

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.

### Version

from doris:be-4.0.0 to doris:be-4.1.1

test: apache/doris:fe-4.1.1 / apache/doris:be-4.1.1

Runtime version reported by FE/BE:

```text
doris-4.1.1-rc01-b10073ad9ca
```

Official source tag/commit matched from Docker BE log:

```text
4.1.1
b10073ad9ca17cd5685c4dd3b3ef650f256376d0
```

Test environment:

```text
Host architecture: arm64
Docker client: 28.5.2
Docker server: 29.5.2
FE image: apache/doris:fe-4.1.1, linux/arm64, sha256:318ab41551d884ded601366193d6115ffdf6471e78e28c572c3dfcfa99d2255e
BE image: apache/doris:be-4.1.1, linux/arm64, sha256:4905607a194641fb47284b616836766aca283ae85575491c799351a41deec60d
```

### What's Wrong?

A minimal query using nested lambda expressions crashes Doris BE with SIGSEGV.

The query does not require any physical table or inserted data. It only builds a one-element array with `array_agg('a')`, then evaluates `array_map(x -> array_count(y -> y = x, ids), ids)`.

Minimal crashing query:

```sql
WITH base AS (
SELECT array_agg('a') AS ids
)
SELECT array_map(
x -> array_count(y -> y = x, ids),
ids
) AS result
FROM base;
```

Expected result:

```text
[1]
```

Actual client error:

```text
ERROR 1105 (HY000) at line 1: RpcException, msg: send fragments failed. io.grpc.StatusRuntimeException: UNAVAILABLE: io exception, host: 172.30.82.3
```

The BE process exits. `docker ps -a` shows:

```text
doris-minrepro-be Exited (0) 9 seconds ago apache/doris:be-4.1.1
```

BE log shows the failing query and SIGSEGV:

```text
*** Query id: f944e19b5ee14d86-bfd49529a73f9a6a ***
*** is nereids: 1 ***
*** tablet id: 0 ***
*** Aborted at 1779956843 (unix time) try "date -d @1779956843" if you are using GNU date ***
*** Current BE git commitID: b10073ad9ca ***
*** SIGSEGV address not mapped to object (@0x0) received by PID 762 (TID 1215 OR 0xfffde7f865c0) from PID 0; stack trace: ***
```

Relevant stack frames:

```text
doris::signal::(anonymous namespace)::FailureSignalHandler
doris::is_column_const
doris::PreparedFunctionImpl::default_implementation_for_constant_arguments
doris::VectorizedFnCall::_do_execute
doris::VLambdaFunctionExpr::execute_column
doris::ArrayMapFunction::execute
doris::VectorizedFnCall::_do_execute
doris::VLambdaFunctionExpr::execute_column
doris::ArrayMapFunction::execute
doris::PipelineTask::execute
```

This is not only a crash. Nested lambda binding also appears semantically wrong even when the query does not crash:

```sql
SELECT array_map(x -> array_count(y -> y = x, ['a']), ['b']) AS should_be_zero;
SELECT array_map(x -> array_count(y -> y = x, ['a']), ['a']) AS should_be_one;
```

Expected:

```text
should_be_zero
[0]
should_be_one
[1]
```

Actual on 4.1.1:

```text
should_be_zero
[1]
should_be_one
[1]
```

So the root issue seems to be nested lambda scoping/capture, and the `array_agg` version turns the same scoping bug into an invalid column access and BE crash.

### What You Expected?

The minimal crashing query should return `[1]` and should never crash BE.

Nested lambdas should resolve variables according to lexical lambda scope. For example:

```sql
SELECT array_map(x -> array_count(y -> y = x, ['a']), ['b']);
```

should return `[0]`, because the outer lambda variable `x` is `'b'`, while the inner array contains only `'a'`.

BE should also fail safely with a query error if an expression binding is invalid. A SQL expression should not be able to terminate the BE process.

### How to Reproduce?

Run a minimal FE/BE Docker cluster:

```bash
docker rm -f doris-minrepro-fe doris-minrepro-be >/dev/null 2>&1 || true
docker network rm doris-minrepro >/dev/null 2>&1 || true

docker network create --subnet 172.30.82.0/24 doris-minrepro

docker run -d \
--name doris-minrepro-fe \
--network doris-minrepro \
--ip 172.30.82.2 \
-p 39030:9030 \
-p 38030:8030 \
-e FE_SERVERS=fe1:172.30.82.2:9010 \
-e FE_ID=1 \
apache/doris:fe-4.1.1

for i in $(seq 1 90); do
if docker exec doris-minrepro-fe mysql -uroot -h127.0.0.1 -P9030 -e 'SHOW FRONTENDS'; then
break
fi
sleep 2
done

docker run -d \
--name doris-minrepro-be \
--network doris-minrepro \
--ip 172.30.82.3 \
-p 38040:8040 \
-e FE_MASTER_IP=172.30.82.2 \
-e BE_IP=172.30.82.3 \
-e BE_PORT=9050 \
apache/doris:be-4.1.1

for i in $(seq 1 120); do
docker exec doris-minrepro-fe mysql -uroot -h127.0.0.1 -P9030 -e 'SHOW BACKENDS' > /tmp/doris-minrepro-backends.out 2>&1 || true
if grep -q 'true' /tmp/doris-minrepro-backends.out; then
cat /tmp/doris-minrepro-backends.out
break
fi
sleep 2
done
```

Run normal control queries first:

```bash
docker exec -i doris-minrepro-fe mysql -uroot -h127.0.0.1 -P9030 <<'SQL'
WITH base AS (SELECT array_agg('a') AS ids)
SELECT ids FROM base;

WITH base AS (SELECT array_agg('a') AS ids)
SELECT array_count(y -> y = 'a', ids) AS result FROM base;

WITH base AS (SELECT array_agg('a') AS ids)
SELECT array_map(x -> x, ids) AS result FROM base;

SELECT array_map(x -> array_count(y -> y = x, ['a']), ['a']) AS expected_result;
SQL
```

Expected and observed output:

```text
ids
["a"]
result
1
result
["a"]
expected_result
[1]
```

Run the crashing query:

```bash
docker exec -i doris-minrepro-fe mysql -uroot -h127.0.0.1 -P9030 <<'SQL'
WITH base AS (
SELECT array_agg('a') AS ids
)
SELECT array_map(
x -> array_count(y -> y = x, ids),
ids
) AS result
FROM base;
SQL
```

Observed:

```text
ERROR 1105 (HY000) at line 1: RpcException, msg: send fragments failed. io.grpc.StatusRuntimeException: UNAVAILABLE: io exception, host: 172.30.82.3
```

Collect evidence:

```bash
docker ps -a --filter name=doris-minrepro-be --format '{{.Names}}\t{{.Status}}\t{{.Image}}'

docker logs doris-minrepro-be 2>&1 | \
grep -En 'Query id|SIGSEGV|FailureSignalHandler|is_column_const|VectorizedFnCall|VLambdaFunctionExpr|ArrayMapFunction|PipelineTask'

rm -rf /tmp/doris-minrepro-log
docker cp doris-minrepro-be:/opt/apache-doris/be/log /tmp/doris-minrepro-log
```

Optional physical-table reproduction with one row:

```sql
CREATE DATABASE IF NOT EXISTS repro;
USE repro;

DROP TABLE IF EXISTS t_array_lambda;

CREATE TABLE t_array_lambda (
es_id VARCHAR(8),
fid VARCHAR(8)
)
DISTRIBUTED BY HASH(es_id) BUCKETS 1
PROPERTIES (
"replication_num" = "1"
);

INSERT INTO t_array_lambda VALUES ('g', 'a');

WITH base AS (
SELECT es_id, array_agg(fid) AS ids
FROM t_array_lambda
GROUP BY es_id
)
SELECT array_map(
x -> array_count(y -> y = x, ids),
ids
) AS result
FROM base;
```

### Anything Else?

### Source-level investigation

The Docker BE log reports commit `b10073ad9ca`, which matches the official `4.1.1` tag target:

```bash
git ls-remote https://github.com/apache/doris.git | grep 'refs/tags/4.1.1'
# 73552015e7587a04f857cb0257fc6e178958d389 refs/tags/4.1.1
# b10073ad9ca17cd5685c4dd3b3ef650f256376d0 refs/tags/4.1.1^{}
```

From source inspection, the likely root cause is in `ArrayMapFunction`'s handling of `VColumnRef` gaps for nested lambdas.

`ArrayMapFunction::execute` collects slot refs from the lambda body, computes `gap`, then recursively applies that gap to all `VColumnRef` nodes under the body:

https://github.com/apache/doris/blob/b10073ad9ca17cd5685c4dd3b3ef650f256376d0/be/src/exprs/lambda_function/varray_map_function.cpp#L82-L93

```cpp
_collect_slot_ref_column_id(children[0], output_slot_ref_indexs);

if (!output_slot_ref_indexs.empty()) {
auto max_id = std::max_element(output_slot_ref_indexs.begin(), output_slot_ref_indexs.end());
gap = *max_id + 1;
_set_column_ref_column_id(children[0], gap);
}
```

The recursive setter does not appear to stop at nested lambda boundaries:

https://github.com/apache/doris/blob/b10073ad9ca17cd5685c4dd3b3ef650f256376d0/be/src/exprs/lambda_function/varray_map_function.cpp#L339-L347

```cpp
void _set_column_ref_column_id(VExprSPtr expr, int gap) const {
for (const auto& child : expr->children()) {
if (child->is_column_ref()) {
auto* ref = static_cast(child.get());
ref->set_gap(gap);
} else {
_set_column_ref_column_id(child, gap);
}
}
}
```

`VColumnRef::set_gap` only sets the gap once, so an inner lambda variable can inherit the outer lambda gap and cannot be corrected by the inner `ArrayMapFunction` execution:

https://github.com/apache/doris/blob/b10073ad9ca17cd5685c4dd3b3ef650f256376d0/be/src/exprs/vcolumn_ref.h#L78-L82

At execution time, `VColumnRef` reads the column by `column_id + gap`:

https://github.com/apache/doris/blob/b10073ad9ca17cd5685c4dd3b3ef650f256376d0/be/src/exprs/vcolumn_ref.h#L59-L69

The const overload of `Block::get_by_position` has no runtime boundary check:

https://github.com/apache/doris/blob/b10073ad9ca17cd5685c4dd3b3ef650f256376d0/be/src/core/block/block.h#L127-L132

The wrong/out-of-range `ColumnPtr` is then dereferenced in the constant-argument path:

- `PreparedFunctionImpl::default_implementation_for_constant_arguments`: https://github.com/apache/doris/blob/b10073ad9ca17cd5685c4dd3b3ef650f256376d0/be/src/exprs/function/function.cpp#L122-L141
- `VectorizedUtils::all_arguments_are_constant`: https://github.com/apache/doris/blob/b10073ad9ca17cd5685c4dd3b3ef650f256376d0/be/src/exec/common/util.hpp#L156-L163
- `is_column_const`: https://github.com/apache/doris/blob/b10073ad9ca17cd5685c4dd3b3ef650f256376d0/be/src/core/column/column.cpp#L80-L82

This matches the observed crash stack.

### Suggested fix direction

I think the fix should be in lambda variable scoping, not just in the crash site.

1. Make `_set_column_ref_column_id` and `_collect_slot_ref_column_id` scope-aware.
- When traversing the current lambda body, do not recurse into nested `LAMBDA_FUNCTION_EXPR` / nested lambda bodies as if they belonged to the same lambda scope.
- Inner lambda parameters should be assigned/resolved by the inner lambda execution context only.

2. Avoid storing mutable execution-specific gap state directly on shared `VColumnRef` nodes when nested lambda expressions can reuse the same expression tree.
- Passing the gap through an execution context, or cloning/rebinding the relevant lambda body per scope, would be safer than mutating `VColumnRef::_gap` globally.

3. Add a runtime guard in `VColumnRef::execute_column` or use `safe_get_by_position` before dereferencing.
- This would turn the current process crash into a query error.
- However, this is only a safety net. It would not fix the wrong-result case shown above.

4. Add regression tests for both cases:

```sql
-- Should not crash; should return [1]
WITH base AS (
SELECT array_agg('a') AS ids
)
SELECT array_map(
x -> array_count(y -> y = x, ids),
ids
) AS result
FROM base;

-- Should return [0], not [1]
SELECT array_map(x -> array_count(y -> y = x, ['a']), ['b']) AS should_be_zero;
```

### Production context

The production query that first exposed the bug was an account-behavior aggregation using this pattern:

```sql
WITH base AS (
SELECT
es_id,
ARRAY_AGG(fid) AS ids
FROM query_log
WHERE created_at >= '2026-05-18 15:26:00'
AND created_at < '2026-05-18 15:36:00'
GROUP BY es_id
)
SELECT
COUNT(*) AS groups_count,
SUM(ARRAY_SIZE(array_map(
x -> array_count(y -> y = x, ids),
array_distinct(ids)
))) AS bucket_count
FROM base;
```

The minimal reproduction above removes the production table, timestamp filter, grouping cardinality, and data volume from the equation. A single `array_agg('a')` is enough to reproduce the BE crash.

### Are you willing to submit PR?

- [ ] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.