[Python] Reading Hive-style partitioned parquet files from GCS
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Trying to read a spark-generated hive-style partitioned parquet dataset with **gcsfs** and {**}pyarrow{**}, but getting a **FileNotFoundError** if I try to read from the base directory or even if try to read directly from one of the partitions. Not sure if I am doing something wrong or it is not supported.
Note that I have successfully read this hive-style partitioned parquet dataset using other methods to rule out any other issues, including:
- Successful read with pyspark by using spark.read.parquet
- Successful read of a specific partition by passing list of paths to `ParquetDataset`
- Also, tested reading another spark-generated parquet dataset with no Hive-style partitions from GCS and that worked as well.
Below is what I am trying:
```python
import gcsfs
import pyarrow as pa
gcs = gcsfs.GCSFileSystem()
partitions = pa.dataset.partitioning(
pa.schema([("partition_var", pa.string())]), dictionaries="infer", flavor="hive"
)
partitioned_dataset = pa.dataset.dataset(
"path/to/partitioned/dataset/base/dir",
filesystem=gcs,
format="parquet",
partitioning=partitions,
)
partition_of_dataset = pa.dataset.dataset(
"path/to/partitioned/dataset/base/dir/partition_var=some_value",
filesystem=gcs,
format="parquet",
)
```
The errors returned for both are below:
```java
FileNotFoundError: path/to/partitioned/dataset/base/dir/
FileNotFoundError: path/to/partitioned/dataset/base/dir/partition_var=some_value/
```
**Reporter**: [Garrett Weaver](https://issues.apache.org/jira/browse/ARROW-14959)
**Note**: *This issue was originally created as [ARROW-14959](https://issues.apache.org/jira/browse/ARROW-14959). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the Python snippet with gcsfs and pyarrow against the base directory and a single Hive-style partition. Compare those results with the successful path-list read and the non-partitioned GCS dataset; done means the base and partition paths no longer raise FileNotFoundError.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100