apache / apache/datafusion-comet
Spark 4.2: segment-tree window metrics unavailable under CometWindowExec
- Dominant language
- Scala
- Stars
- 1.3k
- Forks
- 373
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 198
Description
Spark 4.2 adds a segment-tree window-aggregation optimization to `WindowExec`, along with metrics such as "number of segment-tree frames prepared" and "number of segment-tree fallback frames prepared", plus an eligibility allowlist.
Comet replaces `WindowExec` with `CometWindowExec`, which has no segment-tree implementation and does not expose those metrics. The new test suites read the metric from a Spark `Window` node in the executed plan and fail with "No Window node in plan" when Comet is enabled.
Affected Spark SQL test suites (currently marked `IgnoreCometSuite` in `dev/diffs/4.2.0.diff`):
- `SegmentTreeWindowMetricsSuite`
- `WindowSegmentTreeAllowlistSuite`
Contributor guide
Research direction
Start by inspecting CometWindowExec and the Spark SQL suites SegmentTreeWindowMetricsSuite and WindowSegmentTreeAllowlistSuite, then review their IgnoreCometSuite entries in dev/diffs/4.2.0.diff. Compare the executed plans and metrics expected by those suites. Done means the affected suites no longer fail with "No Window node in plan" when Comet is enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100