apache / apache/arrow

[R] altrep data type slows down evaluation

Open
#39,004 4 comments 0 reactions 0 assignees View on GitHub
Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

When making evaluations like checking for NA the altrep datatype slows calculation by approx four times. Tested in arrow 10, 12 & 14 on Ubuntu.

```
library(arrow)
library(dplyr)

# generate data
x = runif(29500000) * 10
d = data.frame(cv = x)
write_dataset(d, "/tmp/data.arrow")
# then read back
df = open_dataset("/tmp/data.arrow/") %>% select(cv) %>% collect()
x = df$cv
y = x + 0

identical(x, y)
microbenchmark::microbenchmark(x={sum(is.na(x))}, y={sum(is.na(y))})
```

Results:

Unit: milliseconds
| expr | min | lq | mean | median | uq | max neval |
|---|---|---|---|---|---|---|
|x | 291.8 | 302.2 | 348.8 | 310.2 | 348.8 | 754.8 | 100 |
| y | 85.3 | 87.2 | 108.8 | 89.3 | 133.4 | 225.4 | 100 |

With thanks to Barry for the reprex https://mastodon.scot/@geospacedman@mastodon.social/111450704657241188

### Component(s)

R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.