[Python] Add `exclude_invalid_files` to `ParquetDataset`
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
Add a `exclude_invalid_files` parameter to `ParquetDataset`.
I want to read Parquet files from a bucket that contains a JSON metadata file. Because `ParquetDataset` doesn't expose `exclude_invalid_files`, I get an error:
```
File "/Users/balaji/Documents/GitHub/ray/python/ray/data/datasource/parquet_datasource.py", line 204, in __init__
pq_ds = pq.ParquetDataset(
^^^^^^^^^^^^^^^^^^
File "/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/parquet/core.py", line 1776, in __new__
return _ParquetDatasetV2(
^^^^^^^^^^^^^^^^^^
File "/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/parquet/core.py", line 2490, in __init__
self._dataset = ds.dataset(path_or_paths, filesystem=filesystem,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/dataset.py", line 763, in dataset
return _filesystem_dataset(source, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/dataset.py", line 456, in _filesystem_dataset
return factory.finish(schema)
^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_dataset.pyx", line 2752, in pyarrow._dataset.DatasetFactory.finish
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Error creating dataset. Could not read schema from 'ray-example-data/iris.json'. Is this a 'parquet' file?: Could not open Parquet input source 'ray-example-data/iris.json': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```
This issue is motivated by https://github.com/ray-project/ray/issues/36753.
### Component(s)
Python
Contributor guide
Research direction
Start with the ParquetDataset constructor referenced in pyarrow/parquet/core.py and compare it with the call site in ray/data/datasource/parquet_datasource.py. Add the requested option so JSON files can be ignored when reading a bucket, and verify that the reported ArrowInvalid case is covered by the relevant Python tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100