apache / apache/arrow

[Python] Allow `pyarrow.compute.cast` to coerce errors to null values

Open
#34,976 3 comments 0 reactions 0 assignees View on GitHub
Component: Python Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

I have large array consisting of string data. Unfortunately, there is numerical data mixed with categorical data. `pyarrow` seems to offer no straightforward way to separate them.

```python
import pyarrow as pa

arr = pa.array(["3", "+5", "-4.2", "1,000.00", "foo", "7e-3"], type="string")
print(pa.compute.utf8_is_numeric(arr)) # ynnnnn
pa.compute.cast(arr, pa.float32(), safe=False) # ArrowInvalid: Failed to parse string: 'foo' as a scalar of type float
```

basically, it would be great to have either (or both)

- function that returns boolean mask whether string can be cast to float
- add option to `pyarrow.compute.cast` that replaces errors with null values.

My current workaround is to use cast to pandas: `pd.to_numeric(pd.Series(arr, dtype="string[pyarrow]"), errors="coerce")`.

### Component(s)

Python

Contributor guide

Open the contributing guide

Research direction

Start with the pa.compute.cast entry point and reproduce the mixed string example from the issue. Compare the requested boolean-mask and error-to-null alternatives, then verify that the selected behavior preserves successful conversions while representing unparseable values as null.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.