[Python] Allow `pyarrow.compute.cast` to coerce errors to null values
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the enhancement requested
I have large array consisting of string data. Unfortunately, there is numerical data mixed with categorical data. `pyarrow` seems to offer no straightforward way to separate them.
```python
import pyarrow as pa
arr = pa.array(["3", "+5", "-4.2", "1,000.00", "foo", "7e-3"], type="string")
print(pa.compute.utf8_is_numeric(arr)) # ynnnnn
pa.compute.cast(arr, pa.float32(), safe=False) # ArrowInvalid: Failed to parse string: 'foo' as a scalar of type float
```
basically, it would be great to have either (or both)
- function that returns boolean mask whether string can be cast to float
- add option to `pyarrow.compute.cast` that replaces errors with null values.
My current workaround is to use cast to pandas: `pd.to_numeric(pd.Series(arr, dtype="string[pyarrow]"), errors="coerce")`.
### Component(s)
Python
Contributor guide
Research direction
Start with the pa.compute.cast entry point and reproduce the mixed string example from the issue. Compare the requested boolean-mask and error-to-null alternatives, then verify that the selected behavior preserves successful conversions while representing unparseable values as null.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100