apache / apache/arrow

[C++][Dataset] Provide more robust handling of comparison guarantees in the presence of implicit casts

Open
#27,433 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Priority: Medium Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

After ARROW-8919 it's possible that a field reference may be wrapped in an implicit cast, which complicates destructuring during expression simplification. In particular, some errors can arise as a result of assuming that casts will preserve numeric ordering:

```c++

int two_28 = 1 << 28;

auto partition_expr = less_equal(field_ref("i32"), literal(two_28 + 1));

auto filter = greater(
cast(field_ref("i32"), float32()),
literal(float(two_28)));
```

Currently the RHS of the filter and the partition expression will be considered equal since casting `two_28+1` to float results in the same value as casting `two_28` to float (due to limited FP precision). Since `x <= y` and `x > y` are disjoint, the partition will skipped entirely including any rows where `i32 == two_28+1` (which **should** be selected by this filter).

**Reporter**: [Ben Kietzman](https://issues.apache.org/jira/browse/ARROW-11562) / @bkietz

**Note**: *This issue was originally created as [ARROW-11562](https://issues.apache.org/jira/browse/ARROW-11562). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the C++ example in the issue and tracing expression simplification and destructuring around the implicit-cast behavior introduced by ARROW-8919. The fix is done when the partition expression is not treated as equal to the filter merely because floating-point casts collapse distinct integer values, and rows with i32 == two_28+1 remain selectable.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.