futureverse / futureverse/future.p2p
A workaround for clusters?
- Dominant language
- R
- Stars
- 16
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
`future.p2p` looks intriguing. Communicating through storage seems like it simplifies the interprocess communication. (I had considered that very same storage-based todo/done pattern in the earliest days of `crew`, along with Redis, before ultimately choosing `mirai`.) And since shared storage usually available in some form, the idea of using it to delegate computation to collaborators is promising.
At my workplace, we chafe at the networking security restrictions on our cluster. In particular, reverse tunneling is not allowed, which means we cannot spawn SGE jobs from code running on our laptops. I wonder if {future.p2p} can alleviate pain in these types of situations. We use mounted network drives all the time as a means of sharing storage, and unless there is a specific prohibition against pico.sh, it should be allowed as an intermediary. Combined with nested futures, I wonder if this can give the Futureverse its own equivalent of the [`clustermq` SSH connector](https://mschubert.github.io/clustermq/articles/userguide.html#ssh-connector). I am curious if this is a pattern you are considering.
Contributor guide
No contributing guide indexed for this repository
Research direction
No file, test, or entry point is named. Start by reading the future.p2p documentation and current implementation for storage-based communication, then investigate whether mounted network drives, nested futures, SGE jobs, and pico.sh can support the proposed cluster workflow. Done criteria are not defined in the issue.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100