TPC-DS query 72 slow on modest (10) scale factors
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
When running TPC-DS q72, I've noticed that regardless of the underlying file format, latency increases dramatically even with relatively modest scale factors like 10. I've measured the query at around 2.4 seconds with SF=1, but over 60s when SF=10.
When running in my benchmarking setup, the plan is (as you can see - its extremely join heavy) [here](https://gist.github.com/AdamGS/cea5816b321ca70323975c05d6048f36).
Profiling the query using samply (This is with `branch-50` over parquet, SF=1):
By playing around with it, seems like most time is spent in the loop inside the `chain_traverse` macro. I've tried a few common performance techniques - making it an explicitly inlined generic function, changing how the indices and values memory is managed/written to, but nothing made a noticeable difference.
Contributor guide
Research direction
Start with the linked query plan and the samply profile for TPC-DS q72 over Parquet at SF=1, then inspect the loop inside the chain_traverse macro. Reproduce the SF=1 versus SF=10 slowdown and use profiling and benchmark results to verify that the bottleneck is addressed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100