apache / apache/datafusion-comet

[EPIC] Optimize native scalar expressions used in TPC-DS

Open
#4,936 0 comments 0 reactions 1 assignee Claimed by @andygrove View on GitHub
area:expressions enhancement EPIC performance
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

## Background

This is an umbrella issue for performance-tuning the native scalar expression implementations in `datafusion-comet-spark-expr` (`native/spark-expr/`), prioritizing the expressions that carry the most per-row cost in TPC-DS.

The methodology is documented in the contributor guide (Optimizing Scalar Expressions): benchmark-first with criterion, keep output bit-identical to `main` (existing unit tests are the correctness gate), cover the input shapes where fast paths regress (no-null / sparse-null / dense-null, short / long), and do not submit a change that meaningfully regresses any shape. Tuned expressions are recorded as `Performance (tuned ...)` entries in the per-expression audit pages.

## Scope

TPC-DS per-row scalar work is dominated by decimal arithmetic, casts, and a handful of string/conditional expressions. Aggregates (`sum`/`avg`/`count`/`stddev_samp`) are the biggest overall cost but are out of scope here (aggregate, not scalar).

Candidate expressions (to be confirmed and benchmarked individually):

- [ ] `CheckOverflow` (`math_funcs/internal/checkoverflow.rs`) - wraps the result of every decimal `+ - * /`, `sum`, and `avg`. The common non-ANSI path always allocates a new Decimal128 array via `null_if_overflow_precision`, even when nothing overflows. Opportunity: a no-overflow zero-copy fast path. No benchmark exists yet.
- [ ] Decimal rescale (`math_funcs/internal/decimal_rescale_check.rs`) - used by decimal-to-decimal casts; audit for per-row allocation.
- [ ] Further casts used heavily in TPC-DS (decimal casts), triaged against work already merged or in flight.

## Already optimized / in flight

Many scalar expressions have already been tuned in a prior campaign (casts, `substring`, `date_trunc`, `floor`/`ceil`, `to_json`, `parse_url`, `size`, `unhex`, and others). New work under this epic should check the per-expression audit pages first to avoid re-treading tuned expressions.

## How to contribute

Each expression is a small, self-contained PR:
1. Add a criterion benchmark under `native/spark-expr/benches/` covering the shapes above.
2. Capture a baseline on `main`, apply the optimization, confirm bit-identical output, re-measure.
3. Only submit a strict improvement (no meaningful regression on any shape).
4. Record a `Performance (tuned ...)` entry in the relevant audit page.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.