apache / apache/arrow

[R][C++] "negative buffer resize" error with arrow and dplyr in R

Open
#39,912 22 comments 2 reactions 0 assignees View on GitHub
Component: C++ Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Hi everyone,

I was working on a large dataset with over 1 billion observations, stored in 3040 parquet files, with 41 variables. I read the data with `open_dataset()` and then wanted to apply `dplyr` functions:

```
individual_positions %>%
group_by(user_id) %>%
summarize(n_positions = n()) %>%
count(n_positions, sort = TRUE) %>%
collect()
```

`individual_positions` is my dataset, which consists of different job positions a user held throughout her career. I tried to understand the distribution of the number of all job positions that a user ever held. And I got the following error message:

```
Error in `compute.arrow_dplyr_query()`:
! Invalid: Negative buffer resize: -2147483584
Backtrace:
1. ... %>% collect()
3. arrow:::collect.arrow_dplyr_query(.)
4. arrow:::compute.arrow_dplyr_query(x)
```

I googled what "negative buffer resize" really means, but it was in vain. Can anyone please help me with the interpretation and provide any solutions? I know it's possible to process the dataset in `SAS`, but I'm an R lover and I really want to stick with it. Thanks a lot!

Important update here: Because the observations seemed to be randomly sliced into each parquet file (i.e., a user's position-level observations may be in different parquet files), I think when performing the `group_by()` functions, it has to pull all parquet files together, instead of picking only the necessary few ones. This might be overwhelming for the memory. Do I have to repartition the data? Thanks!

### Component(s)

R, C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.