Run / test Datafusion with JSON Bench from ClickHouse
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
@dentiny (and maybe @onlyjackfrost?) pointed me at a new benchmark from ClickHouse, related to processing JSON files: https://github.com/ClickHouse/JSONBench
It would be great to figure out how to get DataFusion represented in that benchmark / show its performance of processing JSON files
### Describe the solution you'd like
Figure out how to run datafusion on the JSONBench test
1. Integrate it locally into `bench.sh`
2. Figure out how to submit results to the JSON bench repo
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by reading the JSONBench repository and its benchmark instructions, then inspect DataFusion's existing bench.sh integration points. Determine how DataFusion can be run against the JSONBench test and how results are submitted to JSONBench. Done means a local bench.sh integration works and the submission procedure is established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, shell
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100