[VL] Add E2E spill-to-disk benchmark
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
Should add some E2E benchmark code measuring on Velox's spill-to-disk performance in Gluten's code base. For more precise reflection to production use case, it's preferred to add the benchmark code as a Spark benchmark suite.
Contributor guide
Research direction
Start by locating the existing Spark benchmark suites in the Gluten code base and reviewing how they model production-like workloads. Use the Spark benchmark suite entry point to determine how Velox spill-to-disk execution is measured; done means an E2E benchmark reports that performance clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100