[C++] Add cast option to return null for values that can't convert
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
I am importing a dataset with arrow, and then converting variable types. But I got an error message because the `arrow` implementation of `as.integer` can't handle empty strings (which is legal in base R). Is this a bug?
```r
#In R
'' %>% as.integer()
[1] NA
#in arrow
q <- data.table(x=c('','1','2'))
q %>% write_dataset('q')
q2 <- 'q' %>% open_dataset %>% mutate(x=as.integer(x)) %>% collect
Error in `collect()`:
! Invalid: Failed to parse string: '' as a scalar of type int32
Run `rlang::last_error()` to see where the error occurred.
```
Update: tryed to preprocess x with `ifelse` but it also did not work.
```r
'q' %>% open_dataset %>% mutate(x= ifelse(x=='',NA,x)) %>% mutate(x=as.integer(x)) %>% collect
Error in `collect()`:
! NotImplemented: Function 'if_else' has no kernel matching input types (bool, bool, string)
Run `rlang::last_error()` to see where the error occurred.
```
**Reporter**: [Lucas Mation](https://issues.apache.org/jira/browse/ARROW-18241) / @lucasmation
#### Related issues:
- [[R] Possible bug in Handling Blank Conversion to Missing Value](https://github.com/apache/arrow/issues/33446) (is duplicated by)
**Note**: *This issue was originally created as [ARROW-18241](https://issues.apache.org/jira/browse/ARROW-18241). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*
Contributor guide
Research direction
Start by reproducing the R pipeline using open_dataset, mutate, as.integer, and collect with an empty string. Trace the R cast entry point into the C++ conversion behavior and determine how an option could return null for values that cannot convert. Done means the requested cast option handles the empty string as null without breaking existing conversions, with coverage for the shown example.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100