feat: add GroupColumn support for List / Struct / Map / FixedSizeList in multi-column GROUP BY
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
When a `GROUP BY` includes any column with a nested type (`List`, `LargeList`, `FixedSizeList`, `Struct`, `Map`), `multi_group_by::supported_type` returns false for that column and the whole grouping falls back to the byte-encoded `GroupValuesRows` path via `arrow_row::Rows`, even if every other column would be supported by `GroupValuesColumn`. Today's `supported_type` ([source](https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/aggregates/group_values/multi_group_by/mod.rs#L1213-L1239)):
```rust
fn supported_type(data_type: &DataType) -> bool {
matches!(*data_type,
DataType::Int8 | Int16 | ... | Float64
| Decimal128(_, _)
| Utf8 | LargeUtf8 | Utf8View
| Binary | LargeBinary | BinaryView
| Date32 | Date64 | Time32(_) | Timestamp(_, _)
| Boolean
)
}
```
### Impact
A wide multi-column `GROUP BY` where most columns are individually supported by `GroupValuesColumn` (Utf8 / Date / Boolean / Float) but one column is a nested type (e.g. `LargeList>`) is forced onto the slow `arrow_row::Rows` path. Group key memory grows materially compared to the equivalent column-format storage, and per-group comparisons become byte-by-byte rather than vectorized.
### Describe the solution you'd like
Add `GroupColumn` implementations for the missing nested types and extend `supported_type` accordingly. Equality is well-defined by Arrow's `make_equal`-style comparators; hashing of nested values has existing arrow utilities.
Suggested incremental order, easiest first:
1. `FixedSizeList` where `T` is a `GroupColumn`-supported primitive (fixed-width per row, simplest)
2. `List` and `LargeList` with primitive child (variable-length, single offset buffer)
3. `Struct<...>` with all fields being `GroupColumn`-supported (composition of existing columns under a single null mask)
4. `Map` (most complex due to entry ordering semantics)
Each can be its own PR. The existing `arrow_row::Rows` path remains the fallback for any nested type not yet supported, so this is purely an optimization.
### Describe alternatives you've considered
- **arrow_row fallback (current state)**: correct, but significantly more memory and CPU than `GroupValuesColumn` when the grouping mostly consists of cheap columns and the nested column is one of many.
- **Avoid grouping by the nested column at the SQL level**: works case-by-case (e.g. wrap in `ANY_VALUE`) but does not help when the nested column is the legitimate group key.
### Additional context
Issue #22526 is a separate, related pathology on the emit side. This issue is about the input/intern side (which `GroupValues` impl is selected). The two are independent and can land in either order.
Contributor guide
Assessment
This issue has not been assessed yet.