apache / apache/arrow

[R] v11.0.0.2 extremely slow with parquet files written in v9.0.0

Open
#34,319 11 comments 1 reaction 0 assignees View on GitHub
Component: R Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 18h
Merged PRs (30d)
91

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I run pretty straight forward usage of `read_parquet` in several workflows at work. This has been a tremendous upgrade from using rds files with the same content for all the reasons arrow exists. Some of our datasets are wider than longer, which isnt great for parquet, but we usually only need a small subset.

I updated to arrow v11.0.0.2 yesterday and noticed that the identical workflows are loading files 50-60x slower.

A benchmark are selectively choosing 200 columns from a (3400 rows x 16000 column dataset, each column containing two text attributes)
~300ms with 9.0.0
~17000ms using v11.0.0.2.

Ubuntu 8-core intel i7

If I save the file I read from v9 saving with v11, then re-read with v11, it's the same , ~17 seconds

On the mac (M1 10-core) it's about 12 seconds.

I understand there are some significant changes under the hood with c++ under the v10 updates but based on the update notes no action would be required for doing anything fancy with ubuntu (I have gcc v9.4) or mac (clang v13).

Any idea what's going on? We store thousands of parquet files so wondering if we should avoid >= v11.

Sorry if this is mislabeled as a bug, but it seems like one to me as I havent changed anything else but systems meet the requirements.

here's a reprex:
```
library(arrow)
library(microbenchmark)

# fake data, 3500x16000 with attributes
test_data <- as.data.frame(
lapply(setNames(1:16000, paste0("col", 1:16000)), function(x) {
col <- sample(c(1:5, NA), 3500, replace = TRUE)
levels(col) <- paste("level", 1:4)
attr(col, "text") <- "test question text"
col
})
)

# write out the file somewhere
file <- "~/Desktop/test_data_v9.parquet"

### using arrow 9.0.0
packageVersion('arrow')
arrow::write_parquet(test_data, file)

### benchmark
microbenchmark::microbenchmark(
arrow::read_parquet(file, col_select = 1:200), times = 10
)

### NOW SWITCH to arrow 11.0.0.2 just from the 9.0.0 data
packageVersion('arrow')
microbenchmark::microbenchmark(
arrow::read_parquet(file, col_select = 1:200), times = 10
)

### now re-create, save, re-read using all v11
packageVersion('arrow')
file2 <- "~/Desktop/test_data_v11.parquet"
arrow::write_parquet(test_data, file2)

### benchmark
microbenchmark::microbenchmark(
arrow::read_parquet(file2, col_select = 1:200), times = 10
)

```

### Component(s)

R

Contributor guide

Open the contributing guide

Research direction

Start with the provided R reprex and reproduce the `read_parquet` benchmark using the 9.0.0 and 11.0.0.2 versions, including both existing and newly written files. Trace the regression from the R `read_parquet` path into the Arrow C++ implementation. Done means the cause is confirmed and selective reads no longer show the reported slowdown, or the issue has a documented resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, r
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.