apache / apache/arrow

Relax pyarrow.compute.is_in type requirement

Open
#34,982 2 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the enhancement requested

In the following example:

```
>>> a = pa.array([1,2,3], type=pa.int8())
>>> pa.compute.is_in(a, pa.array([255], type=pa.int64()))
Traceback (most recent call last):
File "", line 1, in
File "C:\ps_0310_is\lib\site-packages\pyarrow\compute.py", line 256, in wrapper
return func.call(args, options, memory_pool)
File "pyarrow\_compute.pyx", line 355, in pyarrow._compute.Function.call
File "pyarrow\error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
File "pyarrow\error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Integer value 255 not in range: -128 to 127
>>>
```

It would be nice if this could return false instead of raising an error, given that its impossible for 255 to be in an int8 array. This would be more in line with isin in Pandas as well:

```
>>> a = pd.Series([1,2,3], dtype="int8")
>>> a.isin([255])
0 False
1 False
2 False
dtype: bool
>>>
```

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the pyarrow.compute.is_in example with an int8 array and an int64 value of 255, then trace the is_in entry point and its type validation. Done means out-of-range values return false instead of raising while normal membership checks continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.