apache / apache/auron

Support native last / last(ignoreNulls) aggregate

Open
#2,358 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

## Background

Auron currently accelerates `first` / `first(ignoreNulls)` aggregates natively (`AggFirst` / `AggFirstIgnoresNull`), but `last` / `last(ignoreNulls)` are not implemented natively. As a result they fall back to the generic UDAF path (a JNI call back into the JVM), losing vectorized acceleration.

## Proposal

Add native `last` aggregate support, mirroring the existing `first` implementation:

1. Implement `AggLast` and `AggLastIgnoresNull` in `datafusion-ext-plans` with the same columnar accumulator layout as `first`, but with "later value wins" semantics (every visited row overwrites; on merge the later partial state wins).
2. Wire the two new functions through the `AggFunction` enum, `create_agg`, the protobuf contract (`LAST` / `LAST_IGNORES_NULL`), the `protobuf::AggFunction -> AggFunction` conversion, and the window-aggregate mapping.
3. Add the `Last` expression conversion in `NativeConverters.convertAggregateExpr` (dispatching to `LAST` / `LAST_IGNORES_NULL` by `ignoreNulls`).
4. Declare the native aggregate buffer schema for `Last` in `NativeAggBase.computeNativeAggBufferDataTypes` (`[dataType]` for ignoreNulls, `[dataType, Boolean]` otherwise) so the partial -> shuffle -> final buffer schema matches the native side.

## Scope / Non-goals

- Covers `last(col)` and `last(col, ignoreNulls = true)` over primitive / boolean / string / binary / complex (scalar-value) types, consistent with `first`.
- Window aggregate (`last` over a window) reuses the same `AggFunction`.
- Semantics follow Spark: `last` is order-sensitive and non-deterministic without an explicit ordering (same as `first`).

## Tests

- Rust unit test `agg_exec::test::test_agg_last`: partial -> final two-phase aggregation over a nullable column, verifying `last` (keeps the last visited row including null) and `last(ignoreNulls)` (keeps the last non-null value).
- Scala end-to-end test in `AuronDataFrameAggregateSuite` ("native last / last(ignoreNulls) aggregate", spark34 + spark35): a grouped aggregate exercising the full partial -> shuffle -> final native path, asserting correct values and that the plan offloads to `NativeAggBase`.

Contributor guide

Open the contributing guide

Research direction

Start with the existing AggFirst and AggFirstIgnoresNull implementations in datafusion-ext-plans, then trace AggFunction, create_agg, the protobuf conversion, and window mapping. Update NativeConverters.convertAggregateExpr and NativeAggBase.computeNativeAggBufferDataTypes, then run agg_exec::test::test_agg_last and the named AuronDataFrameAggregateSuite cases; done means both native paths pass with the expected plan offload.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, scala, spark
Domain
backend, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.