futureverse / futureverse/future
Memory/CPU Limitations?
- Dominant language
- R
- Stars
- 1k
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
I am trying to run a job in parallel on a relatively large Windows machine on AWS (488 Gb of RAM), using future. However I am not able to use all available cores, due to memory limitations. I have been able to run the job to completion when I only use two workers, however if I use as many as five, RStudio crashes ([APPCRASH](https://www.deskdecode.com/how-to-fix-appcrash-some-programs-has-stopped-working/)). I set the future plan up as follows:
```
cl <- makeCluster(2)
plan(cluster, workers = cl)
```
and I am using `future_lapply` from the `future.apply` package to execute the code.
Just to confirm, is this behavior most likely due to memory or CPU limitations as I suspect? Or would you expect something else to happen if each worker was running out of memory?
Thanks,
Jonathan
Contributor guide
Assessment
This issue has not been assessed yet.