[Python] pyarrow.parquet.read_table either returns FileNotFound or ArrowInvalid
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
running below results in `"GetFileIno() yielded path 'myBucket/features/MyParquet.parquet/year=2022/part-0019.snappy.parquet' which is outside base dir 'gs://myBucket/features/MyParquet.parquet/' "`
```Java
import pyarrow.parquet as pq
import gcsfs
file_path="gs://myBucket/features/MyParquet.parquet/"
fs=gcsfs.GCSFileSystem()
table=pq.read_table(file_path,filesystem=fs)
```
Removing the gs:// from file_path results in a `FileNotFoundError`. Any variation of / or // at the beginning of the path gives me the 'outside base dir' error.
I also ran the below and got valid results using both file_path patterns, so I know it finds the path just fine.
```Java
from pyarrow.fs import FileSelector, PyFileSystem, FSSpecHandler
filesys = PyFileSystem(FSSpecHandler(fs))
selector = FileSelector(file_path, recursive=True)
filesys.get_file_info(selector)
```
**Environment**: GCP JupyterLab notebooks
**Reporter**: [Callista Rogers](https://issues.apache.org/jira/browse/ARROW-15910)
**Note**: *This issue was originally created as [ARROW-15910](https://issues.apache.org/jira/browse/ARROW-15910). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Reproduce the two path variants in a GCP JupyterLab environment using pyarrow.parquet.read_table, gcsfs.GCSFileSystem, and the PyFileSystem/FSSpecHandler path shown in the report. Compare how read_table resolves the selector and filesystem; done means reading the partitioned Parquet dataset from the gs:// path without FileNotFoundError or an outside-base-dir ArrowInvalid error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100