[R] Hangs on `arrow::open_dataset()` with `dplyr::summarise()`
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
### Describe the bug, including details regarding any error messages, version, and platform.
The R session will hang upon failure. This issue appears to be intermittent, it fails around 1 in 10 times with the current settings.
Windows Server 2019
RStudio 2023.03.1
R 4.1.3
Arrow 13.0.0
dplyr 1.1.2
Here is the minimum working example, adjust `vec_length` to refine the failure rate:
```
# create synthetic dataframe of integers (larger dfs are more likely to fail)
vec_length <- 10000000
test_data <- data.frame(test_values = sample(0:10, vec_length, replace = TRUE))
# path to temporarily store the test_dataset
test_dataset_path <- file.path(getwd(), "test_data")
# write to parquet arrow dataset (single file)
arrow::write_dataset(
dataset = test_data,
path = test_dataset_path,
format = "parquet"
)
# attempt to summarise the arrow dataset
for (i in 1:100) {
print(paste("Attempt", i, "to read and summarise test arrow dataset..."))
largest_value <-
arrow::open_dataset(test_dataset_path) |>
dplyr::summarise(max_value = max(test_values)) |>
dplyr::collect()
print("Success.")
}
# remove test data
unlink(test_dataset_path)
```
### Component(s)
R
Contributor guide
Assessment
This issue has not been assessed yet.