futureverse / futureverse/future.p2p

A workaround for clusters?

Open
#2 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
R
Stars
16
Forks
1
PR merge metrics
No merged PRs in 30d

Description

`future.p2p` looks intriguing. Communicating through storage seems like it simplifies the interprocess communication. (I had considered that very same storage-based todo/done pattern in the earliest days of `crew`, along with Redis, before ultimately choosing `mirai`.) And since shared storage usually available in some form, the idea of using it to delegate computation to collaborators is promising.

At my workplace, we chafe at the networking security restrictions on our cluster. In particular, reverse tunneling is not allowed, which means we cannot spawn SGE jobs from code running on our laptops. I wonder if {future.p2p} can alleviate pain in these types of situations. We use mounted network drives all the time as a means of sharing storage, and unless there is a specific prohibition against pico.sh, it should be allowed as an intermediary. Combined with nested futures, I wonder if this can give the Futureverse its own equivalent of the [`clustermq` SSH connector](https://mschubert.github.io/clustermq/articles/userguide.html#ssh-connector). I am curious if this is a pattern you are considering.

Contributor guide

No contributing guide indexed for this repository

Research direction

No file, test, or entry point is named. Start by reading the future.p2p documentation and current implementation for storage-based communication, then investigate whether mounted network drives, nested futures, SGE jobs, and pico.sh can support the proposed cluster workflow. Done criteria are not defined in the issue.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.