[R] altrep data type slows down evaluation
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
When making evaluations like checking for NA the altrep datatype slows calculation by approx four times. Tested in arrow 10, 12 & 14 on Ubuntu.
```
library(arrow)
library(dplyr)
# generate data
x = runif(29500000) * 10
d = data.frame(cv = x)
write_dataset(d, "/tmp/data.arrow")
# then read back
df = open_dataset("/tmp/data.arrow/") %>% select(cv) %>% collect()
x = df$cv
y = x + 0
identical(x, y)
microbenchmark::microbenchmark(x={sum(is.na(x))}, y={sum(is.na(y))})
```
Results:
Unit: milliseconds
| expr | min | lq | mean | median | uq | max neval |
|---|---|---|---|---|---|---|
|x | 291.8 | 302.2 | 348.8 | 310.2 | 348.8 | 754.8 | 100 |
| y | 85.3 | 87.2 | 108.8 | 89.3 | 133.4 | 225.4 | 100 |
With thanks to Barry for the reprex https://mastodon.scot/@geospacedman@mastodon.social/111450704657241188
### Component(s)
R
Contributor guide
Assessment
This issue has not been assessed yet.