apache / apache/arrow

[R] Passing a large dataset to duckdb and back results in memory being used and not freed

Open
#37,495 6 comments 0 reactions 0 assignees View on GitHub
Component: R Type: bug
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the bug, including details regarding any error messages, version, and platform.

I'll flesh this out with a proper reprex later; the dataset in question is the data from the Voltron Data S3 bucket

```
nyc_taxi <- open_dataset(here::here("data/nyc-taxi"))

nyc_taxi %>%
filter(year == 2019) %>%
to_duckdb() %>%
mutate(mean_distance = mean(trip_distance)) %>%
to_arrow() %>%
filter(trip_distance < mean_distance) %>%
count() %>%
collect()
```

I'm seeing 22Gb of RAM usage here despite the returned result only being 1 row

### Component(s)

R

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.