apache / apache/arrow

[Python] pyarrow's read_table is way slower than iter_batches

Open
#32,108 6 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

Hi!

Loading a table created from DataFrame  `pyarrow.parquet.read_table()` is taking 3x  much time as loading it as batches

 
```java

pyarrow.Table.from_batches(
list(pyarrow.parquet.ParquetFile.iter_batches()
)
```

#### Minimal example

 
```java

import pandas as pd
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq

df = pd.DataFrame(
    {
        "a": np.random.random(10**9), 
        "b": np.random.random(10**9)
    }
)

df.to_parquet("file.parquet")

table_of_whole_file = pq.read_table("file.parquet")

table_of_batches = pa.Table.from_batches(
    list(
        pq.ParquetFile("file.parquet").iter_batches()
    )
)

table_of_one_batch = pa.Table.from_batches(
    [
        next(pq.ParquetFile("file.parquet")
        .iter_batches(batch_size=10**9))
    ]
)
```
 

_table_of_batches_ reading time is 11.5 seconds, _table_of_whole_file_ read time is 33.2s.

Also loading table as one batch _table_of_one_batch_ is slightly faster: 9.8s.
#### Parquet file metadata

 
```java

created_by: parquet-cpp-arrow version 8.0.0
num_columns: 2
num_rows: 1000000000
num_row_groups: 15
format_version: 1.0
serialized_size: 5680
```
 

 

**Environment**: pyarrow 8.0.0
pandas 1.4.2
numpy 1.22.4
python 3.9

I reproduced this behaviour on two machines: 
* macbook pro with m1 max 32 gb and cpython 3.9.12 from conda miniforge
* pytorch docker container on standard linux machine
**Reporter**: [Satoshi Nakamoto](https://issues.apache.org/jira/browse/ARROW-16775)
#### Original Issue Attachments:
- [image-2022-06-16-03-04-25-158.png](https://issues.apache.org/jira/secure/attachment/13045144/image-2022-06-16-03-04-25-158.png)

**Note**: *This issue was originally created as [ARROW-16775](https://issues.apache.org/jira/browse/ARROW-16775). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the benchmark with pyarrow.parquet.read_table(), ParquetFile.iter_batches(), and the provided pandas and NumPy setup. Trace the Python Parquet reader entry points into the Arrow implementation and compare the two read paths. Done means the performance discrepancy is explained and the relevant behavior is corrected or covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.