apache / apache/arrow

[Python] ArrowInvalid error when trying to read list of s3 paths in pyarrow.dataset

Open
#34,683 4 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Using a recent nightly on Ubuntu 22.04.2.

`pyarrow.dataset.dataset` can load a parquet directory from an `s3`-prefixed URI but throws an error if you pass it a list of individual parquet files that share the same schema.

```python
[ins] In [1]: import pyarrow

[ins] In [2]: pyarrow.__version__
Out[2]: '12.0.0.dev266'

[ins] In [3]: import pyarrow.dataset as ds
/home/gil/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/compute.py:206: RuntimeWarning: Python binding for RunEndEncodeOptions not exposed
warnings.warn("Python binding for {} not exposed"

[ins] In [4]: files = [
...: f"s3://gbif-open-data-us-east-1/occurrence/2023-02-01/occurrence.parquet/00000{i}"
...: for i in range(3)
...: ]

[ins] In [5]: ds.dataset(files)
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[5], line 1
----> 1 ds.dataset(files)

File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:765, in dataset(source, schema, format, filesystem, partitioning, partition_base_dir, exclude_invalid_files, ignore_prefixes)
763 elif isinstance(source, (tuple, list)):
764 if all(_is_path_like(elem) for elem in source):
--> 765 return _filesystem_dataset(source, **kwargs)
766 elif all(isinstance(elem, Dataset) for elem in source):
767 return _union_dataset(source, **kwargs)

File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:443, in _filesystem_dataset(source, schema, filesystem, partitioning, format, partition_base_dir, exclude_invalid_files, selector_ignore_prefixes)
440 partitioning = _ensure_partitioning(partitioning)
442 if isinstance(source, (list, tuple)):
--> 443 fs, paths_or_selector = _ensure_multiple_sources(source, filesystem)
444 else:
445 fs, paths_or_selector = _ensure_single_source(source, filesystem)

File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:351, in _ensure_multiple_sources(paths, filesystem)
344 is_local = (
345 isinstance(filesystem, (LocalFileSystem, _MockFileSystem)) or
346 (isinstance(filesystem, SubTreeFileSystem) and
347 isinstance(filesystem.base_fs, LocalFileSystem))
348 )
350 # allow normalizing irregular paths such as Windows local paths
--> 351 paths = [filesystem.normalize_path(_stringify_path(p)) for p in paths]
353 # validate that all of the paths are pointing to existing *files*
354 # possible improvement is to group the file_infos by type and raise for
355 # multiple paths per error category
356 if is_local:

File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:351, in (.0)
344 is_local = (
345 isinstance(filesystem, (LocalFileSystem, _MockFileSystem)) or
346 (isinstance(filesystem, SubTreeFileSystem) and
347 isinstance(filesystem.base_fs, LocalFileSystem))
348 )
350 # allow normalizing irregular paths such as Windows local paths
--> 351 paths = [filesystem.normalize_path(_stringify_path(p)) for p in paths]
353 # validate that all of the paths are pointing to existing *files*
354 # possible improvement is to group the file_infos by type and raise for
355 # multiple paths per error category
356 if is_local:

File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/_fs.pyx:967, in pyarrow._fs.FileSystem.normalize_path()

File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()

File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()

ArrowInvalid: Expected a local filesystem path, got a URI: 's3://gbif-open-data-us-east-1/occurrence/2023-02-01/occurrence.parquet/000000'
```

You can work around this with union dataset, but this seems like it should work?

workaround:
```
[ins] In [6]: ds.dataset(list(map(ds.dataset, files)))
```

The performance of `ds.dataset` loading files from s3 is _much_ improved compared with Arrow 11 -- for the files above I was seeing a pretty consistent ~4s per-file and that's dropped down by at least 5x on this nightly!

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with pyarrow.dataset.dataset(files), then trace the issue through _filesystem_dataset and _ensure_multiple_sources as shown in the traceback. Compare this with the working union-dataset workaround and add or run coverage for multiple S3 file paths; done means a list of shared-schema S3 files loads without ArrowInvalid.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.