parquet.read_table opens files twice
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
It appears that parquet files are opened twice when using parquet.read_table. The first time seems to be to read in the schema, the second to do the actual data read.
This seems to be the case when use_legacy_dataset=False and without specifying a schema argument. It also is the case for datasets and single parquet reads.
My use case is that i'd like to use a custom fsspec filesystem that interfaces to an s3 like API. It handles the remote download of the parquet file and passes to pyarrow a handle of a temporary file downloaded locally. Without cached access, this behavior results in files being downloaded twice.
[Here](https://github.com/apache/arrow/blob/5a55fb4f66cbd316568085129054b5ff1d2eb089/python/pyarrow/parquet/core.py#L2479) is where the file is opened to read in table schema.
Here is code to reproduce:
```
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
from fsspec.implementations.local import LocalFileSystem
import os
# create a fsspec filesystem to track the number of times files are opened
class LocalFSWithAccessCounter(LocalFileSystem):
def __init__(self, *args, **storage_options):
super().__init__(*args, **storage_options)
self.access_count = {}
def open(
self,
path,
mode="rb",
block_size=None,
cache_options=None,
compression=None,
**kwargs,
):
if path in self.access_count:
self.access_count[path] += 1
else:
self.access_count[path] = 1
return super().open(
path,
mode,
block_size,
cache_options,
compression,
**kwargs,
)
# create a simple test table
df = pd.DataFrame({'col1':range(2),'col2':range(2)})
out_table = pa.Table.from_pandas(df)
# write then read as single parquet file.
# file is opened twice using read_table
pq.write_table(out_table, 'example.parquet')
fs = LocalFSWithAccessCounter()
in_table = pq.read_table('example.parquet', filesystem=fs)
print('files open count:\n', fs.access_count, '\n\n')
#files open count:
# {'example.parquet': 2}
# write then read as dataset with partitions.
# one file is opened twice when using read_table
pq.write_to_dataset(out_table, 'example_ds.parquet', partition_cols=['col1'])
fs = LocalFSWithAccessCounter()
in_table = pq.read_table(os.path.join(os.getcwd(), 'example_ds.parquet'), filesystem=fs)
print('dataset file open count:\n',fs.access_count)
#dataset file open count:
# {'example.parquet': 2, 'example_ds.parquet/col1=0/237b18616a4844b3b094c264f6a09c48-0.parquet': 2, example_ds.parquet/col1=1/2d70ed145ee742e99f4c5528b053446d-0.parquet': 1, }
```
### Component(s)
Python
Contributor guide
Research direction
Start with the linked schema-reading location in python/pyarrow/parquet/core.py and run the provided fsspec access-counter reproducer for both a single file and a dataset. Trace why parquet.read_table opens files twice, then verify the completed change avoids the duplicate open while preserving the reported reads.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100