Tools to benchmark expression performance
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
As gluten has more and more native expressions support, it's better to have a separate tool to stress/test expressions only.
The general idea is:
- using pyarrow to generate some mock data, with respect to the required data type. e.g., valida json, DATE format string
- using pyspark to call the target spark functions
- collect the output, write into parquet/noop format, so spark can iterate the whole records
-yuan
Contributor guide
Research direction
The issue names no files or tests. Start by mapping the proposed pyarrow mock-data generation, PySpark function invocation, and Parquet/no-op output flow; done would be a separate tool that stress-tests expressions across the required data types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100