apache / apache/arrow

[C++][R] Inconsistent application of type in Datasets via the schema

Open
#29,896 5 comments 0 reactions 0 assignees View on GitHub
Component: C++ Component: R Priority: Critical Status: needs champion Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

 

It looks like at least `filter` is not handling a column type specified by `schema `when specified in `open_dataset. `Reprex:
```java

options("max.print" = 5)
library(arrow, warn.conflicts = FALSE)
library(dplyr, warn.conflicts = FALSE)
## Set up the data
tf <- tempfile()
dir.create(tf)
write_dataset(quakes, tf)
## Works as expected
open_dataset(tf) %>%
filter(stations == 41) %>%
collect()
#> lat long depth mag stations
#> 1 -20.42 181.62 562 4.8 41
#> [ reached 'max' / getOption("max.print") -- omitted 11 rows ]
## errors as expected
open_dataset(tf) %>%
filter(stations == "41") %>%
collect()
#> Error: NotImplemented: Function equal has no kernel matching input types (array[int32], scalar[string])
## Ok let's change a column type
tf_reg <- open_dataset(tf)$schema
tf_reg$stations <- string()
## ok returns a character
open_dataset(tf, schema = tf_reg) %>%
pull(stations) %>%
typeof()
#> [1] "character"
## So if `stations` is character I think this should work?
open_dataset(tf, schema = tf_reg) %>%
filter(stations == as.character("41")) %>%
collect()
#> Error: Filter expression not supported for Arrow Datasets: stations == as.character("41")
#> Call collect() first to pull data into R.
## previous behaviour no longer works
open_dataset(tf, schema = tf_reg) %>%
filter(stations == 41) %>%
collect()
#> Error: NotImplemented: Function equal has no kernel matching input types (array[string], scalar[double])
 
```
 

**Reporter**: [Sam Albers](https://issues.apache.org/jira/browse/ARROW-14324) / @boshek

**Note**: *This issue was originally created as [ARROW-14324](https://issues.apache.org/jira/browse/ARROW-14324). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by running the R reprex through open_dataset(), schema, filter(), and collect() to reproduce the type mismatch. Trace how the supplied schema is applied to dataset filter expressions; done means comparisons use the schema-defined column type consistently and the character and numeric examples behave as documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, r
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.