dask / dask/dask-expr

Reading from parquet files with more than one row group triggers expensive statistics collection

Open
#363 27 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

cc @rjzamora

We should turn this off by default and use the logic one file = one partition. Could you take a look? I am not very familiar with the read_parquet stuff.

```
def read_data(filename):
path = "s3://coiled-runtime-ci/tpc-h/scale-100/" + filename + "/"
return dd.read_parquet(path, engine="pyarrow", filesystem=None)

if __name__ == "__main__":
from distributed import Client
client = Client()

var1 = datetime.strptime("1995-01-01", "%Y-%m-%d")
var2 = datetime.strptime("1997-01-01", "%Y-%m-%d")

line_item_ds = read_data("lineitem")

lineitem_filtered = line_item_ds[
(line_item_ds["l_shipdate"] >= var1) & (line_item_ds["l_shipdate"] < var2)
]
lineitem_filtered["l_year"] = 1 # lineitem_filtered["l_shipdate"].dt.year
lineitem_filtered["revenue"] = lineitem_filtered["l_extendedprice"] * (
1.0 - lineitem_filtered["l_discount"]
)

lineitem_filtered.optimize()

```

This is how we found that particular issue

cc @fjetter

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the example through read_data, dd.read_parquet with the PyArrow engine, and lineitem_filtered.optimize(). Trace where reading Parquet datasets with multiple row groups collects statistics and compare it with the requested one-file-one-partition behavior. Done means the default avoids that expensive statistics collection while preserving the filtered computation's expected behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.