apache / apache/arrow

parquet.read_table opens files twice

Open
#35,894 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

It appears that parquet files are opened twice when using parquet.read_table. The first time seems to be to read in the schema, the second to do the actual data read.

This seems to be the case when use_legacy_dataset=False and without specifying a schema argument. It also is the case for datasets and single parquet reads.

My use case is that i'd like to use a custom fsspec filesystem that interfaces to an s3 like API. It handles the remote download of the parquet file and passes to pyarrow a handle of a temporary file downloaded locally. Without cached access, this behavior results in files being downloaded twice.

[Here](https://github.com/apache/arrow/blob/5a55fb4f66cbd316568085129054b5ff1d2eb089/python/pyarrow/parquet/core.py#L2479) is where the file is opened to read in table schema.

Here is code to reproduce:
```
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from fsspec.implementations.local import LocalFileSystem
import os

# create a fsspec filesystem to track the number of times files are opened
class LocalFSWithAccessCounter(LocalFileSystem):
def __init__(self, *args, **storage_options):
super().__init__(*args, **storage_options)
self.access_count = {}

def open(
self,
path,
mode="rb",
block_size=None,
cache_options=None,
compression=None,
**kwargs,
):
if path in self.access_count:
self.access_count[path] += 1
else:
self.access_count[path] = 1

return super().open(
path,
mode,
block_size,
cache_options,
compression,
**kwargs,
)

# create a simple test table
df = pd.DataFrame({'col1':range(2),'col2':range(2)})
out_table = pa.Table.from_pandas(df)

# write then read as single parquet file.
# file is opened twice using read_table
pq.write_table(out_table, 'example.parquet')
fs = LocalFSWithAccessCounter()
in_table = pq.read_table('example.parquet', filesystem=fs)
print('files open count:\n', fs.access_count, '\n\n')
#files open count:
# {'example.parquet': 2}

# write then read as dataset with partitions.
# one file is opened twice when using read_table
pq.write_to_dataset(out_table, 'example_ds.parquet', partition_cols=['col1'])
fs = LocalFSWithAccessCounter()
in_table = pq.read_table(os.path.join(os.getcwd(), 'example_ds.parquet'), filesystem=fs)
print('dataset file open count:\n',fs.access_count)
#dataset file open count:
# {'example.parquet': 2, 'example_ds.parquet/col1=0/237b18616a4844b3b094c264f6a09c48-0.parquet': 2, example_ds.parquet/col1=1/2d70ed145ee742e99f4c5528b053446d-0.parquet': 1, }
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with the linked schema-reading location in python/pyarrow/parquet/core.py and run the provided fsspec access-counter reproducer for both a single file and a dataset. Trace why parquet.read_table opens files twice, then verify the completed change avoids the duplicate open while preserving the reported reads.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.