Improve performance of TPCH q4, q7, q9 at TPCH SF 100
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
@MrPowers reportes in discord: https://discord.com/channels/885562378132000778/1290751484807352412/1407568961561952277
> I ran the TPC-H queries on my Macbook M3 with 16GB of RAM with different scale factors. The DataFusion, DuckDB, and Polars Streaming results are similar for scale factor 5:
> I ran the TPC-H queries on my Macbook M3 with 16GB of RAM with different scale factors. The DataFusion, DuckDB, and Polars Streaming results are similar for scale factor 5:
### Describe the solution you'd like
Figure out why q4, q7 and q9 are very slow
The TPCH queries are here: https://github.com/apache/datafusion/tree/main/benchmarks/queries
[q4](https://github.com/apache/datafusion/blob/main/benchmarks/queries/q4.sql)
```sql
select
o_orderpriority,
count(*) as order_count
from
orders
where
o_orderdate >= '1993-07-01'
and o_orderdate < date '1993-07-01' + interval '3' month
and exists (
select
*
from
lineitem
where
l_orderkey = o_orderkey
and l_commitdate < l_receiptdate
)
group by
o_orderpriority
order by
o_orderpriority;
```
[q7](https://github.com/apache/datafusion/blob/main/benchmarks/queries/q7.sql)
```sql
select
supp_nation,
cust_nation,
l_year,
sum(volume) as revenue
from
(
select
n1.n_name as supp_nation,
n2.n_name as cust_nation,
extract(year from l_shipdate) as l_year,
l_extendedprice * (1 - l_discount) as volume
from
supplier,
lineitem,
orders,
customer,
nation n1,
nation n2
where
s_suppkey = l_suppkey
and o_orderkey = l_orderkey
and c_custkey = o_custkey
and s_nationkey = n1.n_nationkey
and c_nationkey = n2.n_nationkey
and (
(n1.n_name = 'FRANCE' and n2.n_name = 'GERMANY')
or (n1.n_name = 'GERMANY' and n2.n_name = 'FRANCE')
)
and l_shipdate between date '1995-01-01' and date '1996-12-31'
) as shipping
group by
supp_nation,
cust_nation,
l_year
order by
supp_nation,
cust_nation,
l_year;
```
[q9](https://github.com/apache/datafusion/blob/main/benchmarks/queries/q9.sql)
```sql
select
nation,
o_year,
sum(amount) as sum_profit
from
(
select
n_name as nation,
extract(year from o_orderdate) as o_year,
l_extendedprice * (1 - l_discount) - ps_supplycost * l_quantity as amount
from
part,
supplier,
lineitem,
partsupp,
orders,
nation
where
s_suppkey = l_suppkey
and ps_suppkey = l_suppkey
and ps_partkey = l_partkey
and p_partkey = l_partkey
and o_orderkey = l_orderkey
and s_nationkey = n_nationkey
and p_name like '%green%'
) as profit
group by
nation,
o_year
order by
nation,
o_year desc;
```
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the TPCH query files benchmarks/queries/q4.sql, q7.sql, and q9.sql, then run the TPCH benchmarks at scale factor 100 to reproduce the reported slowdown. Compare the three queries' execution behavior and identify why they are slow; done means the cause is understood and their performance is improved and measured.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100