apache / apache/gluten

[VL] Add E2E spill-to-disk benchmark

Open
#8,180 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

Should add some E2E benchmark code measuring on Velox's spill-to-disk performance in Gluten's code base. For more precise reflection to production use case, it's preferred to add the benchmark code as a Spark benchmark suite.

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing Spark benchmark suites in the Gluten code base and reviewing how they model production-like workloads. Use the Spark benchmark suite entry point to determine how Velox spill-to-disk execution is measured; done means an E2E benchmark reports that performance clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data-engineering, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.