apache / apache/arrow

[GCS] Scanning hive-partitioned dataset results in orders of magnitude more network traffic than it should

Open
#33,624 1 comment 4 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I have a hive-partitioned dataset in a Google Cloud Storage bucket. Its size is around 54MB according to `gsutil du` and verified by downloading it and checking locally. However, if I open it with `ds = pyarrow.dataset.dataset("gs://...", partitioning="hive", format="parquet")` and then traverse it with `ds.to_batches()`, it results in multiple GB of inbound network traffic and takes much longer than simply downloading the data.

This is with PyArrow 10.0.1. MacOS 12.6 with Apple M1 CPU.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the report with a hive-partitioned Parquet dataset in Google Cloud Storage using pyarrow.dataset.dataset(...), then traverse it with ds.to_batches(). Compare the observed network traffic and runtime with the dataset's roughly 54 MB size and a direct download; done means scanning no longer causes multi-GB traffic.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cloud, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.