[Python] ArrowInvalid error when trying to read list of s3 paths in pyarrow.dataset
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
Using a recent nightly on Ubuntu 22.04.2.
`pyarrow.dataset.dataset` can load a parquet directory from an `s3`-prefixed URI but throws an error if you pass it a list of individual parquet files that share the same schema.
```python
[ins] In [1]: import pyarrow
[ins] In [2]: pyarrow.__version__
Out[2]: '12.0.0.dev266'
[ins] In [3]: import pyarrow.dataset as ds
/home/gil/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/compute.py:206: RuntimeWarning: Python binding for RunEndEncodeOptions not exposed
warnings.warn("Python binding for {} not exposed"
[ins] In [4]: files = [
...: f"s3://gbif-open-data-us-east-1/occurrence/2023-02-01/occurrence.parquet/00000{i}"
...: for i in range(3)
...: ]
[ins] In [5]: ds.dataset(files)
---------------------------------------------------------------------------
ArrowInvalid Traceback (most recent call last)
Cell In[5], line 1
----> 1 ds.dataset(files)
File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:765, in dataset(source, schema, format, filesystem, partitioning, partition_base_dir, exclude_invalid_files, ignore_prefixes)
763 elif isinstance(source, (tuple, list)):
764 if all(_is_path_like(elem) for elem in source):
--> 765 return _filesystem_dataset(source, **kwargs)
766 elif all(isinstance(elem, Dataset) for elem in source):
767 return _union_dataset(source, **kwargs)
File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:443, in _filesystem_dataset(source, schema, filesystem, partitioning, format, partition_base_dir, exclude_invalid_files, selector_ignore_prefixes)
440 partitioning = _ensure_partitioning(partitioning)
442 if isinstance(source, (list, tuple)):
--> 443 fs, paths_or_selector = _ensure_multiple_sources(source, filesystem)
444 else:
445 fs, paths_or_selector = _ensure_single_source(source, filesystem)
File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:351, in _ensure_multiple_sources(paths, filesystem)
344 is_local = (
345 isinstance(filesystem, (LocalFileSystem, _MockFileSystem)) or
346 (isinstance(filesystem, SubTreeFileSystem) and
347 isinstance(filesystem.base_fs, LocalFileSystem))
348 )
350 # allow normalizing irregular paths such as Windows local paths
--> 351 paths = [filesystem.normalize_path(_stringify_path(p)) for p in paths]
353 # validate that all of the paths are pointing to existing *files*
354 # possible improvement is to group the file_infos by type and raise for
355 # multiple paths per error category
356 if is_local:
File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/dataset.py:351, in (.0)
344 is_local = (
345 isinstance(filesystem, (LocalFileSystem, _MockFileSystem)) or
346 (isinstance(filesystem, SubTreeFileSystem) and
347 isinstance(filesystem.base_fs, LocalFileSystem))
348 )
350 # allow normalizing irregular paths such as Windows local paths
--> 351 paths = [filesystem.normalize_path(_stringify_path(p)) for p in paths]
353 # validate that all of the paths are pointing to existing *files*
354 # possible improvement is to group the file_infos by type and raise for
355 # multiple paths per error category
356 if is_local:
File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/_fs.pyx:967, in pyarrow._fs.FileSystem.normalize_path()
File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/error.pxi:144, in pyarrow.lib.pyarrow_internal_check_status()
File ~/mambaforge/envs/pyarrow_nightly/lib/python3.10/site-packages/pyarrow/error.pxi:100, in pyarrow.lib.check_status()
ArrowInvalid: Expected a local filesystem path, got a URI: 's3://gbif-open-data-us-east-1/occurrence/2023-02-01/occurrence.parquet/000000'
```
You can work around this with union dataset, but this seems like it should work?
workaround:
```
[ins] In [6]: ds.dataset(list(map(ds.dataset, files)))
```
The performance of `ds.dataset` loading files from s3 is _much_ improved compared with Arrow 11 -- for the files above I was seeing a pretty consistent ~4s per-file and that's dropped down by at least 5x on this nightly!
### Component(s)
Python
Contributor guide
Research direction
Reproduce the failure with pyarrow.dataset.dataset(files), then trace the issue through _filesystem_dataset and _ensure_multiple_sources as shown in the traceback. Compare this with the working union-dataset workaround and add or run coverage for multiple S3 file paths; done means a list of shared-schema S3 files loads without ArrowInvalid.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100