apache / apache/arrow

[Python] No limit pushdown for scanning Parquet on Azure

Open
#34,608 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I am accessing a Parquet file on Azure data lake with the following code.
(to make the example reproducible, the example has a file publicly accessible on Azure)

```
import pyarrow.dataset as ds
from adlfs import AzureBlobFileSystem

abfs_public = AzureBlobFileSystem(
account_name="azureopendatastorage")

dataset_public = ds.dataset('az://nyctlc/yellow/puYear=2010/puMonth=1/part-00000-tid-8898858832658823408-a1de80bd-eed3-4d11-b9d4-fa74bfbd47bc-426339-18.c000.snappy.parquet', filesystem=abfs_public)
```

The processing time is the same for accessing the full file or only the first 5 rows:

```
dataset_public.head(5)
# 5min 11s

dataset_public.to_table()
# 5min 30s

dataset_public.scanner().head(5)
# 5min 43s
```
I would expect the time to be less for 5 rows. I am not sure about the difference between .scanner().head() and .head().

Regarding reducing the number of columns:
reducing the number of columns retrieved speeds up the query, but the reduction seems small. For example, filtering to only 2 columns out of 21 reduces the query to 2min 7:

```
dataset_public.scanner(columns=['vendorID','passengerCount']).to_table()
# 2min 7s
```

I would have expected that collecting 10% of the columns ( 2 columns instead of 21 columns) to reduce the time by more than half. Unless there is an overhead to the query for collecting from Azure?

Sources for the code:
* Apache Arrow website: https://arrow.apache.org/docs/python/parquet.html#reading-from-cloud-storage
* ADLFS Github page: https://github.com/fsspec/adlfs

Thank you for the outstanding job on the Arrow library

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Reproduce the public Azure Parquet example using AzureBlobFileSystem and compare dataset.head(5), dataset.scanner().head(5), and dataset.to_table(). Check whether the dataset scanner applies a row limit during remote Parquet reads and measure the effect of selecting two columns. Done means the 5-row scan avoids reading the full file, or the limitation and expected behavior are clearly documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
cloud, data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.