apache / apache/arrow

pyarrow.parquet.read_table fs (filesystem) argument does not work with fsspec.implementations.arrow.ArrowFSWrapper objects

Open
#33,110 1 comment 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

My version of PyArrow=8.0.0. When I attempt to use a PyArrow LocalFileSystem object wrapped with ArrowFSWrapper it results in the following error:

```python

import pyarrow as pa
import pyarrow.parquet as pq
from fsspec.implementations.arrow import ArrowFSWrapper

lfs = pa.fs.LocalFileSystem()
fs = ArrowFSWrapper(lfs)
pat = pq.read_table("some/file/location.parquet", filesystem=fs)
```

```Java

{{OSError Traceback (most recent call last) Cell In [12], line 1 ----> 1 pat = pq.read_table( *2* "some/file/location.parquet", *3* filesystem=fs) File /usr/local/lib/python3.8/dist-packages/pyarrow/parquet/__init__.py:2737, in read_table(source, columns, use_threads, metadata, schema, use_pandas_metadata, memory_map, read_dictionary, filesystem, filters, buffer_size, partitioning, use_legacy_dataset, ignore_prefixes, pre_buffer, coerce_int96_timestamp_unit, decryption_properties) *2730* raise ValueError( *2731* "The 'metadata' keyword is no longer supported with the new " *2732* "datasets-based implementation. Specify " *2733* "'use_legacy_dataset=True' to temporarily recover the old " *2734* "behaviour." *2735* ) *2736* try: -> 2737 dataset = _ParquetDatasetV2( *2738* source, *2739* schema=schema, *2740* filesystem=filesystem, *2741* partitioning=partitioning, *2742* memory_map=memory_map, *2743* read_dictionary=read_dictionary, *2744* buffer_size=buffer_size, *2745* filters=filters, *2746* ignore_prefixes=ignore_prefixes,}}
...
File /usr/local/lib/python3.8/dist-packages/pyarrow/io.pxi:193, in pyarrow.lib.NativeFile.get_random_access_file() File /usr/local/lib/python3.8/dist-packages/pyarrow/io.pxi:222, in pyarrow.lib.NativeFile._assert_seekable() OSError: only valid on seekable files
```

If I instead use just the LocalFileSystem object without the ArrowFSWrapper, it works as expected.

**Environment**: Python 3.8.10
**Reporter**: [Anshul Kanakia](https://issues.apache.org/jira/browse/ARROW-17898)

**Note**: *This issue was originally created as [ARROW-17898](https://issues.apache.org/jira/browse/ARROW-17898). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the supplied Python snippet with pyarrow.parquet.read_table, LocalFileSystem, and fsspec.implementations.arrow.ArrowFSWrapper, then trace how the filesystem argument reaches the dataset implementation. Done means the wrapper works for the example while the unwrapped LocalFileSystem behavior remains working.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.