[FEA] Support nested fields for `filter` in `cudf.read_parquet()`
- Dominant language
- C++
- Stars
- 9.8k
- Forks
- 1.1k
- Avg merge
- 3d 6m
- Merged PRs (30d)
- 278
Description
**Is your feature request related to a problem? Please describe.**
I wish I could use the `filter` argument of `cudf.read_parquet()` on nested columns. The in-progress GeoArrow specification is considering allowing a `struct` coordinate representation which provides out-of-the-box column statistics for the inner `x` and `y`. Linestrings, polygons, and multipolygons involve layers of `list<>` nesting that still produce column statistics for the coordinates that would be nice to use with a bounding box filter (see example below).
**Describe the solution you'd like**
It would be nice if the left-hand side of a filter expression could be a tuple instead of a string to specify a nested field. I imagine the nested field of a list is more complicated here but even a nested struct field would be helpful.
**Describe alternatives you've considered**
The current workaround I've used is to flatten the fields before writing the parquet. This is OK but looses the extension type metadata (e.g., CRS) and doesn't scale to the nested types (e.g., linestring, polygon, multipolygon).
**Additional context**
A small illustration with some test data:
```python
>>> import pyarrow as pa
>>> import pyarrow.parquet as pq
>>>
>>>
>>> xs = pa.array([0.0, 1.0, 2.0, 3.0])
>>> ys = pa.array([1.0, 2.0, 3.0, 4.0])
>>> xys = pa.array([
... {"x": 0.0, "y": 1.0},
... {"x": 1.0, "y": 2.0},
... {"x": 2.0, "y": 3.0},
... {"x": 3.0, "y": 4.0},
... ])
>>> table = pa.table([xs, ys, xys], names=["x", "y", "xy"])
>>> pq.write_table(table, "test.parquet")
>>>
>>> # Works!
>>> bounds = [0.5, 1.5, 2.5, 3.5]
>>> cudf.read_parquet(
... "test.parquet",
... filters=[
... [
... ('x', '>=', bounds[0]),
... ('y', '>=', bounds[1]),
... ('x', '<=', bounds[2]),
... ('y', '<=', bounds[3])
... ]
... ]
... )
x y xy
0 1.0 2.0 {'x': 1.0, 'y': 2.0}
1 2.0 3.0 {'x': 2.0, 'y': 3.0}
>>>
>>> # Doesn't work:
>>> cudf.read_parquet(
... "test.parquet",
... filters=[
... [
... (('xy', 'x'), '>=', bounds[0]),
... (('xy', 'y'), '>=', bounds[1]),
... (('xy', 'x'), '<=', bounds[2]),
... (('xy', 'y'), '<=', bounds[3])
... ]
... ]
... )
.conda/lib/python3.10/site-packages/cudf/io/parquet.py:674: UserWarning: Row-wise filtering failed in read_parquet for [[(('xy', 'x'), '>=', 0.5), (('xy', 'y'), '>=', 1.5), (('xy', 'x'), '<=', 2.5), (('xy', 'y'), '<=', 3.5)]]
warnings.warn(
x y xy
0 0.0 1.0 {'x': 0.0, 'y': 1.0}
1 1.0 2.0 {'x': 1.0, 'y': 2.0}
2 2.0 3.0 {'x': 2.0, 'y': 3.0}
3 3.0 4.0 {'x': 3.0, 'y': 4.0}
```
Contributor guide
Assessment
This issue has not been assessed yet.