dask / dask/dask-expr

Predicate validation in parquet could happen before compute

Open
#753 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
89
Forks
26
PR merge metrics
No merged PRs in 30d

Description

```
ddf = dd.from_dict(
{"A": range(8), "B": [1, 1, 2, 2, 3, 3, 4, 4]},
npartitions=4,
)
ddf.to_parquet(tmp_path, engine=engine)

with pytest.raises(ValueError, match="not a valid operator in predicates"):
unsupported_op = [[("B", "not eq", 1)]]
dd.read_parquet(tmp_path, engine=engine, filters=unsupported_op)
```

Ideally, this would raise before we trigger compute

Contributor guide

Open the contributing guide

Research direction

Start at the dd.read_parquet filters and predicate-validation entry points, then compare them with the ddf.to_parquet setup shown in the example. Confirm that an unsupported operator is rejected before computation begins, using the provided pytest.raises case as the regression test; done means the same ValueError is raised without triggering compute.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.