apache / apache/arrow

[Python] pyarrow.parquet.read_table either returns FileNotFound or ArrowInvalid

Open
#31,339 13 comments 0 reactions 0 assignees View on GitHub
Component: Parquet Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

running below results in `"GetFileIno() yielded path 'myBucket/features/MyParquet.parquet/year=2022/part-0019.snappy.parquet' which is outside base dir 'gs://myBucket/features/MyParquet.parquet/' "`

```Java

import pyarrow.parquet as pq
import gcsfs
file_path="gs://myBucket/features/MyParquet.parquet/"
fs=gcsfs.GCSFileSystem()
table=pq.read_table(file_path,filesystem=fs) 
```
Removing the gs:// from file_path results in a `FileNotFoundError`. Any variation of / or // at the beginning of the path gives me the 'outside base dir' error.

I also ran the below and got valid results using both file_path patterns, so I know it finds the path just fine.

```Java

from pyarrow.fs import FileSelector, PyFileSystem, FSSpecHandler
filesys = PyFileSystem(FSSpecHandler(fs))
selector = FileSelector(file_path, recursive=True)
filesys.get_file_info(selector)
```

**Environment**: GCP JupyterLab notebooks
**Reporter**: [Callista Rogers](https://issues.apache.org/jira/browse/ARROW-15910)

**Note**: *This issue was originally created as [ARROW-15910](https://issues.apache.org/jira/browse/ARROW-15910). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Reproduce the two path variants in a GCP JupyterLab environment using pyarrow.parquet.read_table, gcsfs.GCSFileSystem, and the PyFileSystem/FSSpecHandler path shown in the report. Compare how read_table resolves the selector and filesystem; done means reading the partitioned Parquet dataset from the gs:// path without FileNotFoundError or an outside-base-dir ArrowInvalid error.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.