apache / apache/datafusion-comet

Native `collect_set(struct)` aggregate crashes on spill emit

Open
#5,239 2 comments 0 reactions 0 assignees View on GitHub
area:aggregation bug priority:high
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### Describe the bug

Native `collect_set` / `collect_list` over a struct with a non-nullable nested field aborts in `GroupedHashAggregateStream::emit` during spill:

```shell
ArrowError: column types must match schema types,
expected List(Struct("colA": Boolean, ...))
but found List(Struct("colA": non-null Boolean, ...)) at column index 1
```

Call Stack: GroupedHashAggregateStream::emit (row_hash.rs:1134) ← spill ← try_update_memory_reservation. Fires only on the spill path.

Dependencies versions: DataFusion 54.1.0 / arrow 58.4.0.

### Steps to reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start at row_hash.rs:1134 in GroupedHashAggregateStream::emit and trace the spill path through try_update_memory_reservation. Investigate the schema mismatch between nullable and non-nullable nested struct fields for native collect_set/collect_list using DataFusion 54.1.0 and Arrow 58.4.0. Done means the spill emit no longer aborts and the aggregate preserves the expected schema.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.