apache / apache/datafusion-comet

Support RANGE window frames with an explicit offset on DATE or DECIMAL ORDER BY

Open
#4,834 2 comments 0 reactions 0 assignees View on GitHub
area:expressions enhancement
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Comet accelerates window functions with `RANGE` frames, but falls back to Spark when the frame uses an explicit `PRECEDING`/`FOLLOWING` offset and the `ORDER BY` column is `DATE` or `DECIMAL`.

See the fallbacks in `spark/src/main/scala/org/apache/spark/sql/comet/CometWindowExec.scala` (the RANGE-frame checks around lines 377-391):

- DATE ORDER BY: falls back with "RANGE frame with explicit offset on DATE ORDER BY is not supported" because Date32 + Int32 frame arithmetic is not handled in the native comparator.
- DECIMAL ORDER BY: falls back with "RANGE frame with explicit offset on DECIMAL ORDER BY is not supported" because decimal precision widening breaks the frame comparator.

Pure `UNBOUNDED`/`CURRENT ROW` RANGE frames on these order-by types already run natively; only explicit numeric offsets fall back.

Documented as a current limitation in the window function compatibility guide.

Contributor guide

Open the contributing guide

Research direction

Start with the RANGE-frame checks around lines 377-391 in spark/src/main/scala/org/apache/spark/sql/comet/CometWindowExec.scala and trace the native comparator path for DATE and DECIMAL ORDER BY. Use the window function compatibility guide and existing window-function coverage to verify that explicit PRECEDING/FOLLOWING offsets run natively while UNBOUNDED and CURRENT ROW behavior remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.