Predicate validation in parquet could happen before compute
- Dominant language
- Python
- Stars
- 89
- Forks
- 26
- PR merge metrics
- No merged PRs in 30d
Description
```
ddf = dd.from_dict(
{"A": range(8), "B": [1, 1, 2, 2, 3, 3, 4, 4]},
npartitions=4,
)
ddf.to_parquet(tmp_path, engine=engine)
with pytest.raises(ValueError, match="not a valid operator in predicates"):
unsupported_op = [[("B", "not eq", 1)]]
dd.read_parquet(tmp_path, engine=engine, filters=unsupported_op)
```
Ideally, this would raise before we trigger compute
Contributor guide
Research direction
Start at the dd.read_parquet filters and predicate-validation entry points, then compare them with the ddf.to_parquet setup shown in the example. Confirm that an unsupported operator is rejected before computation begins, using the provided pytest.raises case as the regression test; done means the same ValueError is raised without triggering compute.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100