[R] v11.0.0.2 extremely slow with parquet files written in v9.0.0
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 91
Description
### Describe the bug, including details regarding any error messages, version, and platform.
I run pretty straight forward usage of `read_parquet` in several workflows at work. This has been a tremendous upgrade from using rds files with the same content for all the reasons arrow exists. Some of our datasets are wider than longer, which isnt great for parquet, but we usually only need a small subset.
I updated to arrow v11.0.0.2 yesterday and noticed that the identical workflows are loading files 50-60x slower.
A benchmark are selectively choosing 200 columns from a (3400 rows x 16000 column dataset, each column containing two text attributes)
~300ms with 9.0.0
~17000ms using v11.0.0.2.
Ubuntu 8-core intel i7
If I save the file I read from v9 saving with v11, then re-read with v11, it's the same , ~17 seconds
On the mac (M1 10-core) it's about 12 seconds.
I understand there are some significant changes under the hood with c++ under the v10 updates but based on the update notes no action would be required for doing anything fancy with ubuntu (I have gcc v9.4) or mac (clang v13).
Any idea what's going on? We store thousands of parquet files so wondering if we should avoid >= v11.
Sorry if this is mislabeled as a bug, but it seems like one to me as I havent changed anything else but systems meet the requirements.
here's a reprex:
```
library(arrow)
library(microbenchmark)
# fake data, 3500x16000 with attributes
test_data <- as.data.frame(
lapply(setNames(1:16000, paste0("col", 1:16000)), function(x) {
col <- sample(c(1:5, NA), 3500, replace = TRUE)
levels(col) <- paste("level", 1:4)
attr(col, "text") <- "test question text"
col
})
)
# write out the file somewhere
file <- "~/Desktop/test_data_v9.parquet"
### using arrow 9.0.0
packageVersion('arrow')
arrow::write_parquet(test_data, file)
### benchmark
microbenchmark::microbenchmark(
arrow::read_parquet(file, col_select = 1:200), times = 10
)
### NOW SWITCH to arrow 11.0.0.2 just from the 9.0.0 data
packageVersion('arrow')
microbenchmark::microbenchmark(
arrow::read_parquet(file, col_select = 1:200), times = 10
)
### now re-create, save, re-read using all v11
packageVersion('arrow')
file2 <- "~/Desktop/test_data_v11.parquet"
arrow::write_parquet(test_data, file2)
### benchmark
microbenchmark::microbenchmark(
arrow::read_parquet(file2, col_select = 1:200), times = 10
)
```
### Component(s)
R
Contributor guide
Research direction
Start with the provided R reprex and reproduce the `read_parquet` benchmark using the 9.0.0 and 11.0.0.2 versions, including both existing and newly written files. Trace the regression from the R `read_parquet` path into the Arrow C++ implementation. Done means the cause is confirmed and selective reads no longer show the reported slowdown, or the issue has a documented resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, r
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100