apache / apache/arrow

nulls in input to pc.case_when cause incorrect results when `len(cond) >~ 70`

Open
#38,139 1 comment 0 reactions 0 assignees View on GitHub
Component: Python Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

Issue: `pc.case_when` gives incorrect results when the input `cond` contains null struct values, and the length of `cond` is greater than ~70.

Example:

When the length of `cond` is small (in this case 2), I get what I would expect:

```python
from pyarrow import compute as pc

print(pc.case_when([{'0': True, '1': False}, {'0': None, '1': None}], [0,0], [1,1], [-1,-1]).to_pylist())
# prints [0, -1]
```

But if cond gets longer than about ~70, it starts replacing all values after a certain point with the default value. Here's the same example as above, but 50x longer (100 elements),

```python
from pyarrow import compute as pc

pc.case_when([{'0': True, '1': False}, {'0': None, '1': None}] * 50, [0,0] * 50, [1,1] *50, [-1,-1] * 50).to_pylist()
```

Expected behavior:

Should print

[0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1]

Actual behavior:

Prints (note all the -1s at the end).

[0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, 0, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1, -1]

Pyarrow version: 10.0.1

### Component(s)

Python

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.