futureverse / futureverse/parallelly

as.cluster run asynchronous parallel?

Open
#13 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
140
Forks
9
PR merge metrics
No merged PRs in 30d

Description

Hi there,
Great package! Really cool. I am using the plan function along with as.cluster where docker images are being pulled on the VMs. The issue is that this processes is done sequentially for a list of VMs ... and it seems like it could/should take place asynchronously in parallel... which would save a *lot* of time if you have a long list of VMs (as made using googleComputeEngineR).

Here's what I'm talking about:

```
my_rscript <- c("docker",
"run", c("--net=host","--shm-size=10G"),
"gibbsdavidl/google_r:v3",
"Rscript")

plan(cluster,
workers = as.cluster(vms,
docker_image="gibbsdavidl/google_r:v3",
rscript=my_rscript)
)
```

The as.cluster call pulls the docker image for each VM in order ... one at a time ... and it's really slow. I could probably make a 'skinnier' docker image however.

Thanks!
-dave

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.