apache / apache/arrow

[Python][Dataset] Detect and use _metadata file in a list of file paths

Open
#24,624 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Priority: Major Status: needs champion Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

From https://github.com/dask/dask/pull/6047#discussion_r402391318

When specifying a directory to `ParquetDataset`, we will detect if a `_metadata` file is present in the directory and use that to populate the `metadata` attribute (and not include this file in the list of "pieces", since it does not include any data).

However, when passing a list of files to `ParquetDataset`, with one being "_metadata", the metadata attribute is not populated, and the "_metadata" path is included as one of the ParquetDatasetPiece objects instead (which leads to an ArrowIOError during the read of that piece).

We _could_ detect it in a list of paths as well.

Note, I mentioned `ParquetDataset`, but if working on this, we should probably directly do it in the datasets API-based version.
Also, I labeled this as Python and not C++ for now, as this might be something that can be handled on the Python side (once the C++ side knows how to process this kind of metadata -> ARROW-8062)

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-8446) / @jorisvandenbossche
#### Related issues:
- [[Python][C++][Dataset] Implement split_row_groups for ParquetDataset](https://github.com/apache/arrow/issues/19181) (relates to)
- [[Python][C++] Possibly use `_common_metadata` for schema if `_metadata` isn't available](https://github.com/apache/arrow/issues/18055) (is related to)
- [[Python][C++] Document how to write _metadata, _common_metadata files with Parquet datasets](https://github.com/apache/arrow/issues/19502) (is related to)
- [[C++][Dataset] Parquet Dataset factory from a _metadata/_common_metadata file](https://github.com/apache/arrow/issues/24275) (is related to)

**Note**: *This issue was originally created as [ARROW-8446](https://issues.apache.org/jira/browse/ARROW-8446). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the Python datasets API and the list-of-path handling for Parquet datasets, then compare it with the existing directory handling described in the issue. Done means a list containing _metadata populates the metadata attribute and excludes that path from the ParquetDatasetPiece objects; check whether the C++ metadata support referenced by ARROW-8062 is required.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.