apache / apache/datafusion-comet
[EPIC] Optimize native scalar expressions used in TPC-DS
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
## Background
This is an umbrella issue for performance-tuning the native scalar expression implementations in `datafusion-comet-spark-expr` (`native/spark-expr/`), prioritizing the expressions that carry the most per-row cost in TPC-DS.
The methodology is documented in the contributor guide (Optimizing Scalar Expressions): benchmark-first with criterion, keep output bit-identical to `main` (existing unit tests are the correctness gate), cover the input shapes where fast paths regress (no-null / sparse-null / dense-null, short / long), and do not submit a change that meaningfully regresses any shape. Tuned expressions are recorded as `Performance (tuned ...)` entries in the per-expression audit pages.
## Scope
TPC-DS per-row scalar work is dominated by decimal arithmetic, casts, and a handful of string/conditional expressions. Aggregates (`sum`/`avg`/`count`/`stddev_samp`) are the biggest overall cost but are out of scope here (aggregate, not scalar).
Candidate expressions (to be confirmed and benchmarked individually):
- [ ] `CheckOverflow` (`math_funcs/internal/checkoverflow.rs`) - wraps the result of every decimal `+ - * /`, `sum`, and `avg`. The common non-ANSI path always allocates a new Decimal128 array via `null_if_overflow_precision`, even when nothing overflows. Opportunity: a no-overflow zero-copy fast path. No benchmark exists yet.
- [ ] Decimal rescale (`math_funcs/internal/decimal_rescale_check.rs`) - used by decimal-to-decimal casts; audit for per-row allocation.
- [ ] Further casts used heavily in TPC-DS (decimal casts), triaged against work already merged or in flight.
## Already optimized / in flight
Many scalar expressions have already been tuned in a prior campaign (casts, `substring`, `date_trunc`, `floor`/`ceil`, `to_json`, `parse_url`, `size`, `unhex`, and others). New work under this epic should check the per-expression audit pages first to avoid re-treading tuned expressions.
## How to contribute
Each expression is a small, self-contained PR:
1. Add a criterion benchmark under `native/spark-expr/benches/` covering the shapes above.
2. Capture a baseline on `main`, apply the optimization, confirm bit-identical output, re-measure.
3. Only submit a strict improvement (no meaningful regression on any shape).
4. Record a `Performance (tuned ...)` entry in the relevant audit page.
Contributor guide
Assessment
This issue has not been assessed yet.