pyarrow.parquet.read_table fs (filesystem) argument does not work with fsspec.implementations.arrow.ArrowFSWrapper objects
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
My version of PyArrow=8.0.0. When I attempt to use a PyArrow LocalFileSystem object wrapped with ArrowFSWrapper it results in the following error:
```python
import pyarrow as pa
import pyarrow.parquet as pq
from fsspec.implementations.arrow import ArrowFSWrapper
lfs = pa.fs.LocalFileSystem()
fs = ArrowFSWrapper(lfs)
pat = pq.read_table("some/file/location.parquet", filesystem=fs)
```
```Java
{{OSError Traceback (most recent call last) Cell In [12], line 1 ----> 1 pat = pq.read_table( *2* "some/file/location.parquet", *3* filesystem=fs) File /usr/local/lib/python3.8/dist-packages/pyarrow/parquet/__init__.py:2737, in read_table(source, columns, use_threads, metadata, schema, use_pandas_metadata, memory_map, read_dictionary, filesystem, filters, buffer_size, partitioning, use_legacy_dataset, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, decryption_properties) *2730* raise ValueError( *2731* "The 'metadata' keyword is no longer supported with the new " *2732* "datasets-based implementation. Specify " *2733* "'use_legacy_dataset=True' to temporarily recover the old " *2734* "behaviour." *2735* ) *2736* try: -> 2737 dataset = _ParquetDatasetV2( *2738* source, *2739* schema=schema, *2740* filesystem=filesystem, *2741* partitioning=partitioning, *2742* memory_map=memory_map, *2743* read_dictionary=read_dictionary, *2744* buffer_size=buffer_size, *2745* filters=filters, *2746* ignore_prefixes=ignore_prefixes,}}
...
File /usr/local/lib/python3.8/dist-packages/pyarrow/io.pxi:193, in pyarrow.lib.NativeFile.get_random_access_file() File /usr/local/lib/python3.8/dist-packages/pyarrow/io.pxi:222, in pyarrow.lib.NativeFile._assert_seekable() OSError: only valid on seekable files
```
If I instead use just the LocalFileSystem object without the ArrowFSWrapper, it works as expected.
**Environment**: Python 3.8.10
**Reporter**: [Anshul Kanakia](https://issues.apache.org/jira/browse/ARROW-17898)
**Note**: *This issue was originally created as [ARROW-17898](https://issues.apache.org/jira/browse/ARROW-17898). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the supplied Python snippet with pyarrow.parquet.read_table, LocalFileSystem, and fsspec.implementations.arrow.ArrowFSWrapper, then trace how the filesystem argument reaches the dataset implementation. Done means the wrapper works for the example while the unwrapped LocalFileSystem behavior remains working.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100