Reading from parquet files with more than one row group triggers expensive statistics collection
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
cc @rjzamora
We should turn this off by default and use the logic one file = one partition. Could you take a look? I am not very familiar with the read_parquet stuff.
```
def read_data(filename):
path = "s3://coiled-runtime-ci/tpc-h/scale-100/" + filename + "/"
return dd.read_parquet(path, engine="pyarrow", filesystem=None)
if __name__ == "__main__":
from distributed import Client
client = Client()
var1 = datetime.strptime("1995-01-01", "%Y-%m-%d")
var2 = datetime.strptime("1997-01-01", "%Y-%m-%d")
line_item_ds = read_data("lineitem")
lineitem_filtered = line_item_ds[
(line_item_ds["l_shipdate"] >= var1) & (line_item_ds["l_shipdate"] < var2)
]
lineitem_filtered["l_year"] = 1 # lineitem_filtered["l_shipdate"].dt.year
lineitem_filtered["revenue"] = lineitem_filtered["l_extendedprice"] * (
1.0 - lineitem_filtered["l_discount"]
)
lineitem_filtered.optimize()
```
This is how we found that particular issue
cc @fjetter
Contributor guide
Research direction
Start by reproducing the example through read_data, dd.read_parquet with the PyArrow engine, and lineitem_filtered.optimize(). Trace where reading Parquet datasets with multiple row groups collects statistics and compare it with the requested one-file-one-partition behavior. Done means the default avoids that expensive statistics collection while preserving the filtered computation's expected behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100