apache / apache/datafusion-comet

perf: extend field-major processing to nested struct fields

Open
#3,225 3 comments 0 reactions 1 assignee Claimed by @vigneshsiva11 View on GitHub
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 6h
Merged PRs (30d)
190

Description

## Summary

PR #3224 implements field-major processing for struct fields, which moves type dispatch from O(rows × fields) to O(fields). However, for complex nested types (Struct, List, Map inside a struct), it falls back to row-major processing via `append_field`.

This issue tracks extending the field-major optimization to nested Struct fields specifically.

## Current Behavior

In `append_struct_fields_field_major()` (row.rs), complex types fall back to per-row processing:

```rust
// For complex types (struct, list, map), fall back to append_field
// since they have their own nested processing logic
dt @ (DataType::Struct(_) | DataType::List(_) | DataType::Map(_, _)) => {
for (row_idx, i) in (row_start..row_end).enumerate() {
let nested_row = if struct_is_null[row_idx] {
SparkUnsafeRow::default()
} else {
// ... extract nested row
};
append_field(dt, struct_builder, &nested_row, field_idx)?;
}
}
```

This means for deeply nested structs, we lose the benefit of field-major processing at each nesting level.

## Proposed Optimization

For nested Struct fields:
1. Get the nested `StructBuilder` once per field
2. Build nested struct validity in one pass
3. Recursively apply field-major processing to nested struct fields

This would require refactoring to separate validity handling from field value processing.

## Expected Impact

- 1.2-1.5x speedup for workloads with deeply nested struct types
- Benefit multiplies with nesting depth

## Notes

- List and Map fields are harder to optimize due to variable-length elements per row
- This is a follow-up to PR #3224 which implemented the initial field-major optimization

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.