[Python] pyarrow's read_table is way slower than iter_batches
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
Hi!
Loading a table created from DataFrame `pyarrow.parquet.read_table()` is taking 3x much time as loading it as batches
```java
pyarrow.Table.from_batches(
list(pyarrow.parquet.ParquetFile.iter_batches()
)
```
#### Minimal example
```java
import pandas as pd
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
df = pd.DataFrame(
{
"a": np.random.random(10**9),
"b": np.random.random(10**9)
}
)
df.to_parquet("file.parquet")
table_of_whole_file = pq.read_table("file.parquet")
table_of_batches = pa.Table.from_batches(
list(
pq.ParquetFile("file.parquet").iter_batches()
)
)
table_of_one_batch = pa.Table.from_batches(
[
next(pq.ParquetFile("file.parquet")
.iter_batches(batch_size=10**9))
]
)
```
_table_of_batches_ reading time is 11.5 seconds, _table_of_whole_file_ read time is 33.2s.
Also loading table as one batch _table_of_one_batch_ is slightly faster: 9.8s.
#### Parquet file metadata
```java
created_by: parquet-cpp-arrow version 8.0.0
num_columns: 2
num_rows: 1000000000
num_row_groups: 15
format_version: 1.0
serialized_size: 5680
```
**Environment**: pyarrow 8.0.0
pandas 1.4.2
numpy 1.22.4
python 3.9
I reproduced this behaviour on two machines:
* macbook pro with m1 max 32 gb and cpython 3.9.12 from conda miniforge
* pytorch docker container on standard linux machine
**Reporter**: [Satoshi Nakamoto](https://issues.apache.org/jira/browse/ARROW-16775)
#### Original Issue Attachments:
- [image-2022-06-16-03-04-25-158.png](https://issues.apache.org/jira/secure/attachment/13045144/image-2022-06-16-03-04-25-158.png)
**Note**: *This issue was originally created as [ARROW-16775](https://issues.apache.org/jira/browse/ARROW-16775). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the benchmark with pyarrow.parquet.read_table(), ParquetFile.iter_batches(), and the provided pandas and NumPy setup. Trace the Python Parquet reader entry points into the Arrow implementation and compare the two read paths. Done means the performance discrepancy is explained and the relevant behavior is corrected or covered by a regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100