Distribute TPCH tooling used by Datafusion as Docker Images and Github Actions
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
As I was working on https://github.com/apache/datafusion-ray/issues/9, I looked at how tests are run on Datafusion and I discover that the tpch tooling is used to generate TBL and then converted via Rust code [here](https://github.com/edmondop/arrow-datafusion/tree/main/benchmarks/src/tpch). Similar code exists under https://github.com/apache/datafusion-benchmarks/tree/main/tpch , but can't easily be consumed
Since there are an increasing number of repositories in the datafusion ecosystem such as https://github.com/apache/datafusion-python, https://github.com/apache/datafusion-ray, https://github.com/apache/datafusion and https://github.com/apache/datafusion-comet, it would be ideal if these tooling would be easier to consume for the different projects.
### Describe the solution you'd like
Ideally, this tooling should be maintained only once, and repositories should consume them. I have no clear view on where we should maintain this tooling (maybe within the core datafusion itself), but I think we should publish it and make it available downstreams:
- as a docker image
- ideally, also, as a github action
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by comparing benchmarks/src/tpch in DataFusion with the tpch directory in datafusion-benchmarks, then review how the DataFusion, datafusion-python, datafusion-ray, and datafusion-comet repositories currently consume the tooling. Done means the shared TPCH tooling is published as a Docker image and, if feasible, a GitHub Action that downstream repositories can consume.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, rust
- Domain
- ci-cd, devops, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100