apache / apache/arrow

[R] write_to_raw is very slow

Open
#48,908 3 comments 0 reactions 0 assignees View on GitHub
Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I've noticed that serializing an arrow table (an ArrowTabular object) from R using `arrow::write_to_raw` can take about 10x the amount of time that it took to first read in the dataset from disk (just a regular nvme ssd).

Not sure whether this counts as a bug report or a feature enhancement request, but in any case, this seems excessive and currently makes Arrow a no-go for inter-process communication in R, e.g. for parallel processing with the `mirai` package.

Here's a minimal example:

```r
library("arrow")
library("profvis")

data <- data.frame(i = rep(1:10, times=1e5))
for (v in 1:100) {
data[, paste0("v", v)] <- rnorm(1e6)
}

# 790 MiB on disk
write_parquet(data, "sandbox/random.parquet")
file.info("sandbox/random.parquet")$size / 1024 / 1024

profvis({
query <- open_dataset("sandbox/random.parquet")
atbl <- as_arrow_table(query) # 70 ms
tbl <- collect(atbl) # 10 ms
ser <- arrow::write_to_raw(atbl, format = "stream") # 810 ms
# - as.raw.Buffer # (660 ms)
# - write_ipc_stream # (120 ms)
# - buffer # (20 ms)
des <- read_ipc_stream(ser, as_data_frame = FALSE) # 10 ms
})
```

As you can see, it takes 80 ms (70+10) to read the data into R, but 810 ms to serialize it for IPC. `as.raw.Buffer` seems to be the major culprit, but even `write_ipc_stream` takes more time than a full read from disk.

I have observed this same behavior on a 2019 Macbook Air (MacOS, Intel) as well as on a 2025 workstation (Linux, AMD Zen 5). The speed is also similar whether using an R arrow 22.0 binary or a compiled arrow 24... with the latter being maybe a tad faster (600-650 ms instead of 800-810ms) but that could be noise in my benchmark.

For completeness, here are the two package versions I tested with. The binary:

```
Arrow package version: 22.0.0.1

Capabilities:

acero TRUE
dataset TRUE
substrait FALSE
parquet TRUE
json TRUE
s3 TRUE
gcs TRUE
utf8proc TRUE
re2 TRUE
snappy TRUE
gzip TRUE
brotli TRUE
zstd TRUE
lz4 TRUE
lz4_frame TRUE
lzo FALSE
bz2 TRUE
jemalloc TRUE
mimalloc TRUE

Memory:

Allocator mimalloc
Current 5 Gb
Max 5.76 Gb

Runtime:

SIMD Level avx512
Detected SIMD Level avx512

Build:

C++ Library Version 22.0.0
C++ Compiler GNU
C++ Compiler Version 8.3.1
```

... and a version compiled using `install_arrow(nightly = TRUE)`

```r
Arrow package version: 23.0.0.100000000

Capabilities:

acero TRUE
dataset TRUE
substrait FALSE
parquet TRUE
json TRUE
s3 TRUE
gcs TRUE
utf8proc TRUE
re2 TRUE
snappy TRUE
gzip TRUE
brotli TRUE
zstd TRUE
lz4 TRUE
lz4_frame TRUE
lzo FALSE
bz2 TRUE
jemalloc TRUE
mimalloc TRUE

Memory:

Allocator mimalloc
Current 1.76 Gb
Max 1.76 Gb

Runtime:

SIMD Level avx512
Detected SIMD Level avx512

Build:

C++ Library Version 24.0.0-SNAPSHOT
C++ Compiler GNU
C++ Compiler Version 11.4.0
```

As always, thanks for your help, I love arrow/parquet.

### Component(s)

R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.