apache / apache/arrow

[R][C++] Arrow crashes (OOM) on R client with large remote parquet files

Open
#30,594 21 comments 0 reactions 0 assignees View on GitHub
Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

The below should be a reproducible crash:

```java

library(arrow)
library(dplyr)
server <- arrow::s3_bucket("ebird",endpoint_override = "minio.cirrus.carlboettiger.info")

path <- server$path("Oct-2021/observations")
obs <- arrow::open_dataset(path)

path$ls() # observe -- 1 parquet file

obs %>% count() # CRASH

obs %>% to_duckdb() # also crash
```
I have attempted to split this large (~100 GB parquet file) into some smaller files, which helps:

```java

path <- server$path("partitioned")
obs <- arrow::open_dataset(path)
obs$ls() # observe, multiple parquet files now
obs %>% count() 

```
(These parquet files have also been created by arrow, btw, from a single large csv file provided by the original data provider (eBird).  Unfortunately generating the partitioned versions is cumbersome as the data is very unevenly distributed, there's few columns that can avoid creating 1000s of parquet partition files and even so the bulk of the 1-billion rows fall within the same group.  But all the same I think this is a bug as there's no indication why arrow cannot handle a single 100GB parquet file I think?). 

 

Let me know if I can provide more info! I'm testing in R with latest CRAN version of arrow on a machine with 200 GB RAM. 

**Reporter**: [Carl Boettiger](https://issues.apache.org/jira/browse/ARROW-15081) / @cboettig
#### Related issues:
- [[C++][R]Opening a multi-file dataset and writing a re-partitioned version of it fails](https://github.com/apache/arrow/issues/18944) (relates to)

**Note**: *This issue was originally created as [ARROW-15081](https://issues.apache.org/jira/browse/ARROW-15081). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the R example with arrow::open_dataset, count(), and to_duckdb() against the remote 100 GB parquet file, comparing it with the partitioned dataset. Trace the C++ and R paths exercised by those entry points and use the reproduction to define done: both operations complete without an out-of-memory crash on the single large file.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, r
Domain
data-engineering, databases, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.