datafusion-contrib / datafusion-contrib/datafusion-distributed

[EPIC] Add other benchmarking datasets

Open
#629 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
139
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
35

Description

Right now we can run benchmarks on three different datasets:

  • TPCH
  • TPC-DS
  • ClickBench

However, it would be cool to add some more, for example:

  • Variants of existing benchmark, like sorted ClickBench or sorted TPCH.
  • Other datasets, like h2o or imdb.

We have good foundations here for holding more datasets easily, the process typically is:

  1. Create the Rust code that prepares the dataset in https://github.com/datafusion-contrib/datafusion-distributed/tree/main/benchmarks/src under a prepare_<dataset>.rs name.
  2. Create the .sh script that calls the Rust code with the appropriate config and generates the dataset in https://github.com/datafusion-contrib/datafusion-distributed/tree/main/benchmarks under a gen-<dataset>.sh name, under different scale factors or configs:
    • If the dataset allows scale factors, the dataset should be generated in testdata//sf/...all dirs with parquet tables...
    • If not, we can just do testdata//...all dirs with parquet tables...
  3. Place the SQL queries used for benchmarks in https://github.com/datafusion-contrib/datafusion-distributed/tree/main/testdata under <dataset>/queries/q<number>.sql
  4. Wire up the new dataset in https://github.com/datafusion-contrib/datafusion-distributed/blob/main/benchmarks/src/run.rs (queries_for_dataset)

The result is that running a command like this:

WORKERS=8 ./benchmarks/run.sh --threads 2 --dataset <dataset>[/<sf100>]

Should end up benchmarking the new dataset

The task is big and will need to be broken down in several sub issues that people can contribute to independently:

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing dataset preparation files under benchmarks/src, generation scripts under benchmarks, query files under testdata, and queries_for_dataset in benchmarks/src/run.rs. The linked subissues define the independent contributions. Done means a selected dataset is generated in the expected Parquet layout, its queries are present, and benchmarks/run.sh accepts the dataset configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
data-engineering, distributed-systems, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.