[CH] sparkArrayFold aggregate fails with lambda type mismatch on nullable struct inputs
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Backend
CH (Clickhouse)
### Bug description
## Description
`aggregate` (mapped to sparkArrayFold) fails when lambda argument types (accumulator/element) do not exactly match the runtime-captured types, especially with nullable arrays or nullable struct fields. This leads to runtime exceptions or incorrect null results.
## Steps to Reproduce
1. Create a table with array> including null elements/fields.
2. Run aggregate with merge + finish lambda, e.g.:
- merge lambda uses struct accumulator
- finish lambda reads accumulator fields
## Expected Behavior
Results match vanilla Spark.
## Actual Behavior
Runtime error (type mismatch in lambda capture) or incorrect null outputs.
## Proposed Fix
- Align accumulator/element types with merge lambda argument types.
- Cast array elements to the expected lambda element type when needed.
- Add tests for nested struct + nulls.
## Steps to Reproduce
-- setup
```
CREATE TABLE tb_array_complex(items ARRAY>) USING parquet;
INSERT INTO tb_array_complex VALUES
(array(named_struct('v', 1, 'w', 1.5), named_struct('v', null, 'w', 2.0), null)),
(array()),
(null),
(array(named_struct('v', 2, 'w', null), named_struct('v', 3, 'w', 4.5)));
```
-- reproduce
```
SELECT
aggregate(
items,
cast(struct(0 as cnt, 0.0 as sum) as struct),
(acc, x) -> struct(
acc.cnt + if(x is null or x.v is null, 0, 1),
acc.sum + coalesce(x.w, 0.0)
),
acc -> if(acc.cnt = 0, cast(null as double), acc.sum / acc.cnt)
) AS avg_w
FROM tb_array_complex;
```
-- setup
```
CREATE TABLE tb_array_simple(ids ARRAY) USING parquet;
INSERT INTO tb_array_simple VALUES (array(1,5,2,null,3)), (array(1,1,3,2)), (null), (array());
```
-- reproduce
```
SELECT
aggregate(
ids,
cast(struct(0 as cnt, 0.0 as sum) as struct),
(acc, x) -> struct(acc.cnt + 1, acc.sum + coalesce(cast(x as double), 0.0)),
acc -> acc.sum
) AS sum_v
FROM tb_array_simple;
```
### Gluten version
main branch
### Spark version
Spark-3.3.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start at the ClickHouse backend's aggregate implementation mapped to sparkArrayFold, then run the two supplied SQL reproductions with nullable structs, arrays, and lambda captures. Compare the results with vanilla Spark; done means matching results and coverage for nested structs and null elements.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- clickhouse
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100