apache / apache/arrow

[C++][Acero][Python] Asof join does not detect null times

Open
#46,780 1 comment 0 reactions 0 assignees View on GitHub
Component: C++ Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

The current asof join implementation does not check the null bitmap before indexing into the time column's value buffer. This creates matches that should not be possible. For example,

```python
import pyarrow as pa

lhs = pa.table({
"time": pa.array([None], type=pa.int64())
})
rhs = pa.table({
"time": [0],
"data": [True],
})

lhs.join_asof(rhs, "time", [], 0)
```
produces
```
time data
0 null true
```
which implies that null equals 0. By manipulating the value buffer, we can get null to "equal" any integer:
```python
lhs = pa.table({
"time": pa.Array.from_buffers(
pa.int32(),
1,
[pa.py_buffer(b"\x00"), pa.py_buffer(b"\xf0\x00\x00\x00")],
),
})

rhs = pa.table({
"time": pa.array([0xf0], type=pa.int32()),
"payload": ["abc"],
})

# lhs: rhs:
# time time payload
# 0 null 0 240 abc

result = lhs.join_asof(rhs, "time", [], 0)

# result:
# time payload
# 0 null abc -> non-null payload means null matched 240
```

This could be fixed by adding a null check in the `GetTime` function in `time_series_util.cc`, but it's not clear to me what the correct behavior should be.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.