apache / apache/arrow

[Python] Add is_nan, is_null, is_valid as operators to DNF filters

Open
#38,750 2 comments 0 reactions 1 assignee Claimed by @JacekPliszka View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

Currently pyarrow.parquet.core.filters_to_expression handles equality/inequality operators and in and not in operators.

I propose adding is_nan, is_null, is_valid operators where value passed would be ignored but they would
return field.is_nan() field.is_null() field.is_valid() expressions.

This is a very easy change but it would allow null/nan filtering in DNF form.
These functions are already implemented for pyarrow.dataset.Expression https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Expression.html

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.