apache / apache/arrow

[Python] Reading Hive-style partitioned parquet files from GCS

Open
#30,481 7 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Trying to read a spark-generated hive-style partitioned parquet dataset with **gcsfs** and {**}pyarrow{**}, but getting a **FileNotFoundError** if I try to read from the base directory or even if try to read directly from one of the partitions. Not sure if I am doing something wrong or it is not supported.

Note that I have successfully read this hive-style partitioned parquet dataset using other methods to rule out any other issues, including:
- Successful read with pyspark by using spark.read.parquet
- Successful read of a specific partition by passing list of paths to `ParquetDataset`
- Also, tested reading another spark-generated parquet dataset with no Hive-style partitions from GCS and that worked as well.

Below is what I am trying:

 
```python

import gcsfs
import pyarrow as pa
 
gcs = gcsfs.GCSFileSystem()
 
partitions = pa.dataset.partitioning(
    pa.schema([("partition_var", pa.string())]), dictionaries="infer", flavor="hive"
)
 
partitioned_dataset = pa.dataset.dataset(
    "path/to/partitioned/dataset/base/dir",
    filesystem=gcs,
    format="parquet",
    partitioning=partitions,
)
 
partition_of_dataset = pa.dataset.dataset(
    "path/to/partitioned/dataset/base/dir/partition_var=some_value",
    filesystem=gcs,
    format="parquet",
)
```
 

The errors returned for both are below:
```java

FileNotFoundError: path/to/partitioned/dataset/base/dir/
FileNotFoundError: path/to/partitioned/dataset/base/dir/partition_var=some_value/
```
 

**Reporter**: [Garrett Weaver](https://issues.apache.org/jira/browse/ARROW-14959)

**Note**: *This issue was originally created as [ARROW-14959](https://issues.apache.org/jira/browse/ARROW-14959). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Python snippet with gcsfs and pyarrow against the base directory and a single Hive-style partition. Compare those results with the successful path-list read and the non-partitioned GCS dataset; done means the base and partition paths no longer raise FileNotFoundError.

Written by the indexing model from the issue text.

Assessment

Tech stack
gcp, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.