Support native last / last(ignoreNulls) aggregate
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 21
Description
## Background
Auron currently accelerates `first` / `first(ignoreNulls)` aggregates natively (`AggFirst` / `AggFirstIgnoresNull`), but `last` / `last(ignoreNulls)` are not implemented natively. As a result they fall back to the generic UDAF path (a JNI call back into the JVM), losing vectorized acceleration.
## Proposal
Add native `last` aggregate support, mirroring the existing `first` implementation:
1. Implement `AggLast` and `AggLastIgnoresNull` in `datafusion-ext-plans` with the same columnar accumulator layout as `first`, but with "later value wins" semantics (every visited row overwrites; on merge the later partial state wins).
2. Wire the two new functions through the `AggFunction` enum, `create_agg`, the protobuf contract (`LAST` / `LAST_IGNORES_NULL`), the `protobuf::AggFunction -> AggFunction` conversion, and the window-aggregate mapping.
3. Add the `Last` expression conversion in `NativeConverters.convertAggregateExpr` (dispatching to `LAST` / `LAST_IGNORES_NULL` by `ignoreNulls`).
4. Declare the native aggregate buffer schema for `Last` in `NativeAggBase.computeNativeAggBufferDataTypes` (`[dataType]` for ignoreNulls, `[dataType, Boolean]` otherwise) so the partial -> shuffle -> final buffer schema matches the native side.
## Scope / Non-goals
- Covers `last(col)` and `last(col, ignoreNulls = true)` over primitive / boolean / string / binary / complex (scalar-value) types, consistent with `first`.
- Window aggregate (`last` over a window) reuses the same `AggFunction`.
- Semantics follow Spark: `last` is order-sensitive and non-deterministic without an explicit ordering (same as `first`).
## Tests
- Rust unit test `agg_exec::test::test_agg_last`: partial -> final two-phase aggregation over a nullable column, verifying `last` (keeps the last visited row including null) and `last(ignoreNulls)` (keeps the last non-null value).
- Scala end-to-end test in `AuronDataFrameAggregateSuite` ("native last / last(ignoreNulls) aggregate", spark34 + spark35): a grouped aggregate exercising the full partial -> shuffle -> final native path, asserting correct values and that the plan offloads to `NativeAggBase`.
Contributor guide
Research direction
Start with the existing AggFirst and AggFirstIgnoresNull implementations in datafusion-ext-plans, then trace AggFunction, create_agg, the protobuf conversion, and window mapping. Update NativeConverters.convertAggregateExpr and NativeAggBase.computeNativeAggBufferDataTypes, then run agg_exec::test::test_agg_last and the named AuronDataFrameAggregateSuite cases; done means both native paths pass with the expected plan offload.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, scala, spark
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100