futureverse / futureverse/future

How to minimise retransmission of objects when I use SSH connections?

Open
#346 1 comment 0 reactions 0 assignees View on GitHub
Backend API enhancement feature request feature/sticky-globals globals
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

Hello,
I’m using future apply function with ssh connections, cluster plan.
However, future apply function makes retransmission of data frame or objects every mapping procedures even I use same datasets or objects when I’m just changing parameters of estimations to find optimal condition.

Pseudo code is here.

```R
data <- mirt::Science
nFactors <- 1:4
future::plan('cluster', workers = paste0('s',1:2))
future.apply::futute_lapply(X=nFactors, FUN = function(X, data){mirt::mirt(data = data, X)}, data = data)
```

After run this code, let’s watch traffic status. It seems do retransmission of data, even I don’t change any data for the parameter estimation.

How can I reduce data retransmission? That’s make me hard to operate HPC computing on some VPS provider, they makes QoS limit every my calculation.

Best,
Seongho

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.