darwin-eu / darwin-eu/CohortCharacteristics
Running out of memory when calling `summariseCharacteristics()`
- Dominant language
- R
- Stars
- 3
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
The cohort in question is pretty large, and in [this line](https://github.com/darwin-eu/CohortCharacteristics/blob/main/R/summariseCharacteristics.R#L413) the cohort is collected and downloaded to R, throwing the following error:
```r
Error in `purrr::map()`:
ℹ In index: 5.
Caused by error in `dplyr::collect()`:
! Failed to collect lazy table.
Caused by error:
! cannot allocate vector of size 12.5 Gb
```
Is the `collect()` necessary? At least in my use case, the program runs fine without the `collect()`, leaving a lazy table that ` PatientProfiles::summariseResult()` handles just fine.
Contributor guide
Research direction
Start at line 413 of R/summariseCharacteristics.R, where summariseCharacteristics() collects the cohort, and inspect how the result is passed to PatientProfiles::summariseResult(). Reproduce the large-cohort memory failure, then verify that the summarisation completes while preserving the expected result when the cohort remains lazy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100