apache / apache/gluten

[CH] sparkArrayFold aggregate fails with lambda type mismatch on nullable struct inputs

Open
#11,488 0 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Backend

CH (Clickhouse)

### Bug description

## Description
`aggregate` (mapped to sparkArrayFold) fails when lambda argument types (accumulator/element) do not exactly match the runtime-captured types, especially with nullable arrays or nullable struct fields. This leads to runtime exceptions or incorrect null results.

## Steps to Reproduce
1. Create a table with array> including null elements/fields.
2. Run aggregate with merge + finish lambda, e.g.:
- merge lambda uses struct accumulator
- finish lambda reads accumulator fields

## Expected Behavior
Results match vanilla Spark.

## Actual Behavior
Runtime error (type mismatch in lambda capture) or incorrect null outputs.

## Proposed Fix
- Align accumulator/element types with merge lambda argument types.
- Cast array elements to the expected lambda element type when needed.
- Add tests for nested struct + nulls.

## Steps to Reproduce
-- setup
```
CREATE TABLE tb_array_complex(items ARRAY>) USING parquet;

INSERT INTO tb_array_complex VALUES
(array(named_struct('v', 1, 'w', 1.5), named_struct('v', null, 'w', 2.0), null)),
(array()),
(null),
(array(named_struct('v', 2, 'w', null), named_struct('v', 3, 'w', 4.5)));
```

-- reproduce
```
SELECT
aggregate(
items,
cast(struct(0 as cnt, 0.0 as sum) as struct),
(acc, x) -> struct(
acc.cnt + if(x is null or x.v is null, 0, 1),
acc.sum + coalesce(x.w, 0.0)
),
acc -> if(acc.cnt = 0, cast(null as double), acc.sum / acc.cnt)
) AS avg_w
FROM tb_array_complex;
```

-- setup
```
CREATE TABLE tb_array_simple(ids ARRAY) USING parquet;
INSERT INTO tb_array_simple VALUES (array(1,5,2,null,3)), (array(1,1,3,2)), (null), (array());
```

-- reproduce
```
SELECT
aggregate(
ids,
cast(struct(0 as cnt, 0.0 as sum) as struct),
(acc, x) -> struct(acc.cnt + 1, acc.sum + coalesce(cast(x as double), 0.0)),
acc -> acc.sum
) AS sum_v
FROM tb_array_simple;
```

### Gluten version

main branch

### Spark version

Spark-3.3.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start at the ClickHouse backend's aggregate implementation mapped to sparkArrayFold, then run the two supplied SQL reproductions with nullable structs, arrays, and lambda captures. Compare the results with vanilla Spark; done means matching results and coverage for nested structs and null elements.

Written by the indexing model from the issue text.

Assessment

Tech stack
clickhouse
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.