apache / apache/datafusion

TPC-DS query 72 slow on modest (10) scale factors

Open
#17,494 21 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

When running TPC-DS q72, I've noticed that regardless of the underlying file format, latency increases dramatically even with relatively modest scale factors like 10. I've measured the query at around 2.4 seconds with SF=1, but over 60s when SF=10.

When running in my benchmarking setup, the plan is (as you can see - its extremely join heavy) [here](https://gist.github.com/AdamGS/cea5816b321ca70323975c05d6048f36).

Profiling the query using samply (This is with `branch-50` over parquet, SF=1):
Image

By playing around with it, seems like most time is spent in the loop inside the `chain_traverse` macro. I've tried a few common performance techniques - making it an explicitly inlined generic function, changing how the indices and values memory is managed/written to, but nothing made a noticeable difference.

Contributor guide

Open the contributing guide

Research direction

Start with the linked query plan and the samply profile for TPC-DS q72 over Parquet at SF=1, then inspect the loop inside the chain_traverse macro. Reproduce the SF=1 versus SF=10 slowdown and use profiling and benchmark results to verify that the bottleneck is addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.