[Python][Dataset] Detect and use _metadata file in a list of file paths
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
From https://github.com/dask/dask/pull/6047#discussion_r402391318
When specifying a directory to `ParquetDataset`, we will detect if a `_metadata` file is present in the directory and use that to populate the `metadata` attribute (and not include this file in the list of "pieces", since it does not include any data).
However, when passing a list of files to `ParquetDataset`, with one being "_metadata", the metadata attribute is not populated, and the "_metadata" path is included as one of the ParquetDatasetPiece objects instead (which leads to an ArrowIOError during the read of that piece).
We _could_ detect it in a list of paths as well.
Note, I mentioned `ParquetDataset`, but if working on this, we should probably directly do it in the datasets API-based version.
Also, I labeled this as Python and not C++ for now, as this might be something that can be handled on the Python side (once the C++ side knows how to process this kind of metadata -> ARROW-8062)
**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-8446) / @jorisvandenbossche
#### Related issues:
- [[Python][C++][Dataset] Implement split_row_groups for ParquetDataset](https://github.com/apache/arrow/issues/19181) (relates to)
- [[Python][C++] Possibly use `_common_metadata` for schema if `_metadata` isn't available](https://github.com/apache/arrow/issues/18055) (is related to)
- [[Python][C++] Document how to write _metadata, _common_metadata files with Parquet datasets](https://github.com/apache/arrow/issues/19502) (is related to)
- [[C++][Dataset] Parquet Dataset factory from a _metadata/_common_metadata file](https://github.com/apache/arrow/issues/24275) (is related to)
**Note**: *This issue was originally created as [ARROW-8446](https://issues.apache.org/jira/browse/ARROW-8446). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start with the Python datasets API and the list-of-path handling for Parquet datasets, then compare it with the existing directory handling described in the issue. Done means a list containing _metadata populates the metadata attribute and excludes that path from the ParquetDatasetPiece objects; check whether the C++ metadata support referenced by ARROW-8062 is required.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100