futureverse / futureverse/future

Memory/CPU Limitations?

Open
#239 1 comment 0 reactions 0 assignees View on GitHub
feature/resources question
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

Hi there,

I am trying to run a job in parallel on a relatively large Windows machine on AWS (488 Gb of RAM), using future. However I am not able to use all available cores, due to memory limitations. I have been able to run the job to completion when I only use two workers, however if I use as many as five, RStudio crashes ([APPCRASH](https://www.deskdecode.com/how-to-fix-appcrash-some-programs-has-stopped-working/)). I set the future plan up as follows:
```
cl <- makeCluster(2)
plan(cluster, workers = cl)
```
and I am using `future_lapply` from the `future.apply` package to execute the code.

Just to confirm, is this behavior most likely due to memory or CPU limitations as I suspect? Or would you expect something else to happen if each worker was running out of memory?

Thanks,
Jonathan

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.