apache / apache/arrow

[R] Hangs on `arrow::open_dataset()` with `dplyr::summarise()`

Open
#37,798 1 comment 0 reactions 0 assignees View on GitHub
Component: R Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

The R session will hang upon failure. This issue appears to be intermittent, it fails around 1 in 10 times with the current settings.

Windows Server 2019
RStudio 2023.03.1
R 4.1.3
Arrow 13.0.0
dplyr 1.1.2

Here is the minimum working example, adjust `vec_length` to refine the failure rate:

```
# create synthetic dataframe of integers (larger dfs are more likely to fail)
vec_length <- 10000000
test_data <- data.frame(test_values = sample(0:10, vec_length, replace = TRUE))

# path to temporarily store the test_dataset
test_dataset_path <- file.path(getwd(), "test_data")

# write to parquet arrow dataset (single file)
arrow::write_dataset(
dataset = test_data,
path = test_dataset_path,
format = "parquet"
)

# attempt to summarise the arrow dataset
for (i in 1:100) {
print(paste("Attempt", i, "to read and summarise test arrow dataset..."))

largest_value <-
arrow::open_dataset(test_dataset_path) |>
dplyr::summarise(max_value = max(test_values)) |>
dplyr::collect()

print("Success.")
}

# remove test data
unlink(test_dataset_path)
```

### Component(s)

R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.