[C++] Handle nulls consistently in set membership tests (Substrait <--> Acero)
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
There is a TODO here:
https://github.com/apache/arrow/blob/3c013da56fd55122072ffaca3e23afb12c290075/cpp/src/arrow/engine/substrait/expression_internal.cc#L1284
Now that https://github.com/apache/arrow/pull/36739 is merged, can we use the new [`INCONCLUSIVE` null matching option](https://arrow.apache.org/docs/dev/cpp/api/compute.html#_CPPv4N5arrow7compute16SetLookupOptions20NullMatchingBehavior12INCONCLUSIVEE) to translate Substrait [`SingularOrList` expressions](https://github.com/substrait-io/substrait/blob/651829aabde4c9562b291f8bb444aa0303c482cd/proto/substrait/algebra.proto#L1017) to/from Acero `is_in` function options with consistent null handling behavior?
Can we also use this new option to ensure that [Substrait's `index_in` function](https://github.com/substrait-io/substrait/blob/651829aabde4c9562b291f8bb444aa0303c482cd/extensions/functions_set.yaml#L5) is translated to/from Acero's `index_in` function with the correct options set to give consistent behavior?
Some other related issues:
- https://github.com/apache/arrow/issues/36535
- https://github.com/substrait-io/substrait/issues/517
### Component(s)
C++
Contributor guide
Assessment
This issue has not been assessed yet.