apache / apache/arrow

[C++] Add cast option to return null for values that can't convert

Open
#20,486 4 comments 2 reactions 0 assignees View on GitHub
Component: C++ Component: Python Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

I am importing a dataset with arrow, and then converting variable types. But I got an error message because the `arrow` implementation of `as.integer` can't handle empty strings (which is legal in base R). Is this a bug?
```r

#In R
'' %>% as.integer()

[1] NA

 

#in arrow

q <- data.table(x=c('','1','2'))
q %>% write_dataset('q')
q2 <- 'q' %>% open_dataset %>% mutate(x=as.integer(x)) %>% collect

Error in `collect()`:
! Invalid: Failed to parse string: '' as a scalar of type int32
Run `rlang::last_error()` to see where the error occurred.
```
Update: tryed to preprocess x with `ifelse` but it also did not work.
```r

'q' %>% open_dataset %>% mutate(x= ifelse(x=='',NA,x)) %>% mutate(x=as.integer(x)) %>% collect
Error in `collect()`:
! NotImplemented: Function 'if_else' has no kernel matching input types (bool, bool, string)
Run `rlang::last_error()` to see where the error occurred.
```

**Reporter**: [Lucas Mation](https://issues.apache.org/jira/browse/ARROW-18241) / @lucasmation
#### Related issues:
- [[R] Possible bug in Handling Blank Conversion to Missing Value](https://github.com/apache/arrow/issues/33446) (is duplicated by)

**Note**: *This issue was originally created as [ARROW-18241](https://issues.apache.org/jira/browse/ARROW-18241). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the R pipeline using open_dataset, mutate, as.integer, and collect with an empty string. Trace the R cast entry point into the C++ conversion behavior and determine how an option could return null for values that cannot convert. Done means the requested cast option handles the empty string as null without breaking existing conversions, with coverage for the shown example.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.