apache / apache/datafusion-comet

Spark 4.2: segment-tree window metrics unavailable under CometWindowExec

Open
#4,964 0 comments 0 reactions 0 assignees View on GitHub
area:expressions bug priority:low spark 4.2 spark sql tests
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

Spark 4.2 adds a segment-tree window-aggregation optimization to `WindowExec`, along with metrics such as "number of segment-tree frames prepared" and "number of segment-tree fallback frames prepared", plus an eligibility allowlist.

Comet replaces `WindowExec` with `CometWindowExec`, which has no segment-tree implementation and does not expose those metrics. The new test suites read the metric from a Spark `Window` node in the executed plan and fail with "No Window node in plan" when Comet is enabled.

Affected Spark SQL test suites (currently marked `IgnoreCometSuite` in `dev/diffs/4.2.0.diff`):
- `SegmentTreeWindowMetricsSuite`
- `WindowSegmentTreeAllowlistSuite`

Contributor guide

Open the contributing guide

Research direction

Start by inspecting CometWindowExec and the Spark SQL suites SegmentTreeWindowMetricsSuite and WindowSegmentTreeAllowlistSuite, then review their IgnoreCometSuite entries in dev/diffs/4.2.0.diff. Compare the executed plans and metrics expected by those suites. Done means the affected suites no longer fail with "No Window node in plan" when Comet is enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.