apache / apache/datafusion-comet
Native `collect_set(struct)` aggregate crashes on spill emit
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
### Describe the bug
Native `collect_set` / `collect_list` over a struct with a non-nullable nested field aborts in `GroupedHashAggregateStream::emit` during spill:
```shell
ArrowError: column types must match schema types,
expected List(Struct("colA": Boolean, ...))
but found List(Struct("colA": non-null Boolean, ...)) at column index 1
```
Call Stack: GroupedHashAggregateStream::emit (row_hash.rs:1134) ← spill ← try_update_memory_reservation. Fires only on the spill path.
Dependencies versions: DataFusion 54.1.0 / arrow 58.4.0.
### Steps to reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start at row_hash.rs:1134 in GroupedHashAggregateStream::emit and trace the spill path through try_update_memory_reservation. Investigate the schema mismatch between nullable and non-nullable nested struct fields for native collect_set/collect_list using DataFusion 54.1.0 and Arrow 58.4.0. Done means the spill emit no longer aborts and the aggregate preserves the expected schema.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100