[R][C++] "negative buffer resize" error with arrow and dplyr in R
- Dominant language
- C++
- Stars
- 17.1k
- Forks
- 4.3k
- Avg merge
- 3d 13h
- Merged PRs (30d)
- 88
Description
Hi everyone,
I was working on a large dataset with over 1 billion observations, stored in 3040 parquet files, with 41 variables. I read the data with `open_dataset()` and then wanted to apply `dplyr` functions:
```
individual_positions %>%
group_by(user_id) %>%
summarize(n_positions = n()) %>%
count(n_positions, sort = TRUE) %>%
collect()
```
`individual_positions` is my dataset, which consists of different job positions a user held throughout her career. I tried to understand the distribution of the number of all job positions that a user ever held. And I got the following error message:
```
Error in `compute.arrow_dplyr_query()`:
! Invalid: Negative buffer resize: -2147483584
Backtrace:
1. ... %>% collect()
3. arrow:::collect.arrow_dplyr_query(.)
4. arrow:::compute.arrow_dplyr_query(x)
```
I googled what "negative buffer resize" really means, but it was in vain. Can anyone please help me with the interpretation and provide any solutions? I know it's possible to process the dataset in `SAS`, but I'm an R lover and I really want to stick with it. Thanks a lot!
Important update here: Because the observations seemed to be randomly sliced into each parquet file (i.e., a user's position-level observations may be in different parquet files), I think when performing the `group_by()` functions, it has to pull all parquet files together, instead of picking only the necessary few ones. This might be overwhelming for the memory. Do I have to repartition the data? Thanks!
### Component(s)
R, C++
Contributor guide
Assessment
This issue has not been assessed yet.