[Python] No limit pushdown for scanning Parquet on Azure
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I am accessing a Parquet file on Azure data lake with the following code.
(to make the example reproducible, the example has a file publicly accessible on Azure)
```
import pyarrow.dataset as ds
from adlfs import AzureBlobFileSystem
abfs_public = AzureBlobFileSystem(
account_name="azureopendatastorage")
dataset_public = ds.dataset('az://nyctlc/yellow/puYear=2010/puMonth=1/part-00000-tid-8898858832658823408-a1de80bd-eed3-4d11-b9d4-fa74bfbd47bc-426339-18.c000.snappy.parquet', filesystem=abfs_public)
```
The processing time is the same for accessing the full file or only the first 5 rows:
```
dataset_public.head(5)
# 5min 11s
dataset_public.to_table()
# 5min 30s
dataset_public.scanner().head(5)
# 5min 43s
```
I would expect the time to be less for 5 rows. I am not sure about the difference between .scanner().head() and .head().
Regarding reducing the number of columns:
reducing the number of columns retrieved speeds up the query, but the reduction seems small. For example, filtering to only 2 columns out of 21 reduces the query to 2min 7:
```
dataset_public.scanner(columns=['vendorID','passengerCount']).to_table()
# 2min 7s
```
I would have expected that collecting 10% of the columns ( 2 columns instead of 21 columns) to reduce the time by more than half. Unless there is an overhead to the query for collecting from Azure?
Sources for the code:
* Apache Arrow website: https://arrow.apache.org/docs/python/parquet.html#reading-from-cloud-storage
* ADLFS Github page: https://github.com/fsspec/adlfs
Thank you for the outstanding job on the Arrow library
### Component(s)
Python
Contributor guide
Research direction
Reproduce the public Azure Parquet example using AzureBlobFileSystem and compare dataset.head(5), dataset.scanner().head(5), and dataset.to_table(). Check whether the dataset scanner applies a row limit during remote Parquet reads and measure the effect of selecting two columns. Done means the 5-row scan avoids reading the full file, or the limitation and expected behavior are clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- cloud, data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100