duckdb / duckdb/duckdb-python

fetch_record_batch is 2x/3x slower than raw pyArrow

オープン
#359 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
needs triage
主要言語
Python
スター
187
フォーク
112
平均マージ
13時間 29分
マージ済み PR(30日)
17

説明

### What happens?

Hi Team,

First of all, thanks for the all the hard work on DuckDB, it's an amazing product.

From my testing, it seems that `DuckDB` slows down significantly when querying parquet data and serving the result as `record_batches`. I'm not super sure the exact issue but it's usually 2x/3x slower than polars/pyarrow.

You can reproduce it below with `uv run xx.py`
### To Reproduce

```python
# /// script
# requires-python = ">=3.13"
# dependencies = [
# "duckdb==1.4.4",
# "polars==1.38.1",
# "pyarrow==23.0.1",
# ]
# ///
import pyarrow.parquet as pq
import pyarrow as pa

import duckdb
import polars as pl
from itertools import permutations
import time
from contextlib import contextmanager

pa.show_info()

# Create parquet file with permutations of 'abcdef'
perms = list(permutations('abcdefghijk'))
print(len(perms))
data = {'permutation': [''.join(p) for p in perms]}
table = pa.table(data)
pq.write_table(table, 'alphabet_test.parquet')

time_taken = {}
@contextmanager
def timer(name):
start = time.time()
try:
yield
finally:
elapsed = time.time() - start
print(f"{name}: {elapsed:.6f}s")
time_taken[name] = elapsed

# Test DuckDB fetch_arrow
with timer("DuckDB fetch_arrow_table"):
reader = duckdb.read_parquet('alphabet_test.parquet').fetch_arrow_table(batch_size=100_000)

# Test DuckDB fetch_record_batch
with timer("DuckDB fetch_record_batch"):
reader = duckdb.read_parquet('alphabet_test.parquet').fetch_arrow_reader(batch_size=100_000)
batches_duckdb = []
for batch in reader:
batches_duckdb.append(batch)

# Test PyArrow RecordBatchReader
with timer("PyArrow RecordBatchReader"):
ds = pa.dataset.dataset('alphabet_test.parquet')
reader = pa.dataset.Scanner.from_dataset(ds, batch_size=100_000).to_reader()
batches_pyarrow = []
for batch in reader:
batches_pyarrow.append(batch)

# Test Polars + PyArrow RecordBatchReader
with timer("Polars + PyArrow RecordBatchReader"):
ds = pl.scan_parquet('alphabet_test.parquet').collect_batches(chunk_size=100_000)
reader = pa.RecordBatchReader.from_stream(ds)
batches_pyarrow = []
for batch in reader:
batches_pyarrow.append(batch)

print("\nSummary of time taken:")
for name, elapsed in time_taken.items():
print(f"{name}: {elapsed:.6f}s")
print(f"fetch ducdkb batches {time_taken['DuckDB fetch_record_batch'] / time_taken['PyArrow RecordBatchReader']:.2f} slower than PyArrow RecordBatchReader")
```

### OS:

Darwin arm64

### DuckDB Version:

1.4.4

### DuckDB Client:

python

### Hardware:

Apple M4

### Full Name:

Valentino Chen

### Affiliation:

Personal

### Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

- [x] Yes, I have

### Did you include all code required to reproduce the issue?

- [x] Yes, I have

### Did you include all relevant data sets for reproducing the issue?

No - Other reason (please specify in the issue body)

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

まず、指定された DuckDB、PyArrow、Polars のバージョンで、提供されている再現手順を `uv run xx.py` で実行し、`fetch_arrow_table`、`fetch_record_batch`、外部リーダーの実行時間を比較します。Python の record-batch 経路を追跡して、どこで遅延が発生しているかを特定します。ボトルネックが解消され、結果を変更せずにベンチマークでパフォーマンスが向上することが完了条件です。

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
data, performance
issue の種類
バグ
難易度
4/5
見積もり時間
3〜5日
活発さ
停滞
明瞭さ
おおむね明確
初心者へのやさしさ
35/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。