futureverse / futureverse/parallelly
as.cluster run asynchronous parallel?
- Dominant language
- R
- Stars
- 140
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
Hi there,
Great package! Really cool. I am using the plan function along with as.cluster where docker images are being pulled on the VMs. The issue is that this processes is done sequentially for a list of VMs ... and it seems like it could/should take place asynchronously in parallel... which would save a *lot* of time if you have a long list of VMs (as made using googleComputeEngineR).
Here's what I'm talking about:
```
my_rscript <- c("docker",
"run", c("--net=host","--shm-size=10G"),
"gibbsdavidl/google_r:v3",
"Rscript")
plan(cluster,
workers = as.cluster(vms,
docker_image="gibbsdavidl/google_r:v3",
rscript=my_rscript)
)
```
The as.cluster call pulls the docker image for each VM in order ... one at a time ... and it's really slow. I could probably make a 'skinnier' docker image however.
Thanks!
-dave
Contributor guide
Assessment
This issue has not been assessed yet.