apache / apache/datafusion

TPCH q1 with no predicates is 2x slower than duckdb

Open
#18,411 50 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

Breaking out from https://github.com/apache/datafusion/issues/18341#issuecomment-3466350575 from @2010YOUY01

There is a discord discussion for slow tpch q1 https://discord.com/channels/885562378132000778/1290751484807352412/1432863136612089959 from @camuel

```sql
select
l_returnflag,
l_linestatus,
count(*) as count_order
from
lineitem
group by
l_returnflag,
l_linestatus;
```

DuckDB is run by creating views over parquet with this command and also threads are set to 12 (which matches actual cores on my AMD64 machine):
```sql
CREATE VIEW lineitem AS SELECT * FROM read_parquet('...repos/datafusion/sf1000/lineitem/*.parquet');
```

DataFusion is run by running this command:
```
cargo run --profile release --bin tpch -- benchmark datafusion --path ./sf1000 --partitions 12 --format parquet --query 1 --iterations=1 --memory-limit 1G --debug --batch-size 8192 --prefer_hash_join true --mem-pool-type fair
```

gist with logs: https://gist.github.com/camuel/67b4424205b81f06d657ea093ddbfe3c

### To Reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by running the TPCH benchmark command from the issue for query 1 with the supplied sf1000 Parquet data and compare it with the DuckDB setup. Read the linked gist and related discussion to locate the performance gap; done means identifying and addressing the cause so the no-predicate query approaches the expected benchmark performance.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.