[GCS] Scanning hive-partitioned dataset results in orders of magnitude more network traffic than it should
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I have a hive-partitioned dataset in a Google Cloud Storage bucket. Its size is around 54MB according to `gsutil du` and verified by downloading it and checking locally. However, if I open it with `ds = pyarrow.dataset.dataset("gs://...", partitioning="hive", format="parquet")` and then traverse it with `ds.to_batches()`, it results in multiple GB of inbound network traffic and takes much longer than simply downloading the data.
This is with PyArrow 10.0.1. MacOS 12.6 with Apple M1 CPU.
### Component(s)
Python
Contributor guide
Research direction
Reproduce the report with a hive-partitioned Parquet dataset in Google Cloud Storage using pyarrow.dataset.dataset(...), then traverse it with ds.to_batches(). Compare the observed network traffic and runtime with the dataset's roughly 54 MB size and a direct download; done means scanning no longer causes multi-GB traffic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100