apache / apache/arrow

[R] Substantial RAM use increase in 9.0.0 release on write_dataset()

Open
#32,795 12 comments 0 reactions 0 assignees View on GitHub
Component: R Priority: Critical Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

Consider the following example of opening a remote dataset (a single 4 GB parquet file) and streaming it to disk. Consider this reprex:

 
```java

s3 <- arrow::s3_bucket("data", endpoint_override = "minio3.ecoforecast.org", anonymous=TRUE)
df <- arrow::open_dataset(s3$path("waq_test"))
arrow::write_dataset(df, tempfile())
 
```
In 8.0.0, this operation peaks at about ~10 GB RAM use, which is already surprisingly high (when the whole file is 4 GB when on disk), but on arrow 9.0.0 RAM use for the same operation approximately doubles, which is large enough to trigger the OOM killer on the task in several of our active production workflows. 

 

Can this large RAM use increase introduced in 9.0 be avoided?  Is it possible for this operation to use even less RAM than it does in 8.0 release?  Is there something about this particular parquet file that should be responsible for the large RAM use? 

 

Arrow's impressively fast performance on large data on remote hosts is really game-changing for us.  Still, the OOM errors are a bit unexpected at this scale (i.e. single 4GB parquet file), as R users we really depend on arrow's out-of-band operations to work with larger-than-RAM data.

 

**Reporter**: [Carl Boettiger](https://issues.apache.org/jira/browse/ARROW-17541) / @cboettig
#### Related issues:
- [[R] arrow::open_dataset %>% select(myvars) %>% collect causes memory leak](https://github.com/apache/arrow/issues/33366) (is related to)
#### Original Issue Attachments:
- [Screenshot 2022-08-30 at 14-23-20 Online Graph Maker · Plotly Chart Studio.png](https://issues.apache.org/jira/secure/attachment/13048775/Screenshot+2022-08-30+at+14-23-20+Online+Graph+Maker+%C2%B7+Plotly+Chart+Studio.png)

**Note**: *This issue was originally created as [ARROW-17541](https://issues.apache.org/jira/browse/ARROW-17541). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the supplied R reprex using arrow::open_dataset() and arrow::write_dataset() against the remote 4 GB Parquet file, comparing Arrow 8.0.0 and 9.0.0 while measuring peak RAM. The issue names no source files or tests; done means identifying the 9.0.0 regression and establishing that the operation no longer incurs the reported increase, with coverage for the repro if the project has a suitable test location.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.