apache / apache/arrow

[Python] Add `exclude_invalid_files` to `ParquetDataset`

Open
#36,278 5 comments 1 reaction 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Add a `exclude_invalid_files` parameter to `ParquetDataset`.

I want to read Parquet files from a bucket that contains a JSON metadata file. Because `ParquetDataset` doesn't expose `exclude_invalid_files`, I get an error:

```
File "/Users/balaji/Documents/GitHub/ray/python/ray/data/datasource/parquet_datasource.py", line 204, in __init__
pq_ds = pq.ParquetDataset(
^^^^^^^^^^^^^^^^^^
File "/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/parquet/core.py", line 1776, in __new__
return _ParquetDatasetV2(
^^^^^^^^^^^^^^^^^^
File "/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/parquet/core.py", line 2490, in __init__
self._dataset = ds.dataset(path_or_paths, filesystem=filesystem,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/dataset.py", line 763, in dataset
return _filesystem_dataset(source, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/balaji/Documents/GitHub/ray/.venv/lib/python3.11/site-packages/pyarrow/dataset.py", line 456, in _filesystem_dataset
return factory.finish(schema)
^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_dataset.pyx", line 2752, in pyarrow._dataset.DatasetFactory.finish
File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Error creating dataset. Could not read schema from 'ray-example-data/iris.json'. Is this a 'parquet' file?: Could not open Parquet input source 'ray-example-data/iris.json': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.
```

This issue is motivated by https://github.com/ray-project/ray/issues/36753.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with the ParquetDataset constructor referenced in pyarrow/parquet/core.py and compare it with the call site in ray/data/datasource/parquet_datasource.py. Add the requested option so JSON files can be ignored when reading a bucket, and verify that the reported ArrowInvalid case is covered by the relevant Python tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.