TPCH q1 with no predicates is 2x slower than duckdb
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
Breaking out from https://github.com/apache/datafusion/issues/18341#issuecomment-3466350575 from @2010YOUY01
There is a discord discussion for slow tpch q1 https://discord.com/channels/885562378132000778/1290751484807352412/1432863136612089959 from @camuel
```sql
select
l_returnflag,
l_linestatus,
count(*) as count_order
from
lineitem
group by
l_returnflag,
l_linestatus;
```
DuckDB is run by creating views over parquet with this command and also threads are set to 12 (which matches actual cores on my AMD64 machine):
```sql
CREATE VIEW lineitem AS SELECT * FROM read_parquet('...repos/datafusion/sf1000/lineitem/*.parquet');
```
DataFusion is run by running this command:
```
cargo run --profile release --bin tpch -- benchmark datafusion --path ./sf1000 --partitions 12 --format parquet --query 1 --iterations=1 --memory-limit 1G --debug --batch-size 8192 --prefer_hash_join true --mem-pool-type fair
```
gist with logs: https://gist.github.com/camuel/67b4424205b81f06d657ea093ddbfe3c
### To Reproduce
_No response_
### Expected behavior
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start by running the TPCH benchmark command from the issue for query 1 with the supplied sf1000 Parquet data and compare it with the DuckDB setup. Read the linked gist and related discussion to locate the performance gap; done means identifying and addressing the cause so the no-predicate query approaches the expected benchmark performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 32/100