darwin-eu / darwin-eu/CohortCharacteristics

Running out of memory when calling `summariseCharacteristics()`

Open
#6 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
3
Forks
1
PR merge metrics
No merged PRs in 30d

Description

The cohort in question is pretty large, and in [this line](https://github.com/darwin-eu/CohortCharacteristics/blob/main/R/summariseCharacteristics.R#L413) the cohort is collected and downloaded to R, throwing the following error:
```r
Error in `purrr::map()`:
ℹ In index: 5.
Caused by error in `dplyr::collect()`:
! Failed to collect lazy table.
Caused by error:
! cannot allocate vector of size 12.5 Gb
```

Is the `collect()` necessary? At least in my use case, the program runs fine without the `collect()`, leaving a lazy table that ` PatientProfiles::summariseResult()` handles just fine.

Contributor guide

Open the contributing guide

Research direction

Start at line 413 of R/summariseCharacteristics.R, where summariseCharacteristics() collects the cohort, and inspect how the result is passed to PatientProfiles::summariseResult(). Reproduce the large-cohort memory failure, then verify that the summarisation completes while preserving the expected result when the cohort remains lazy.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.