futureverse / futureverse/future

BACKEND: ClusterMQ as a new backend

Open
#204 9 comments 8 reactions 0 assignees View on GitHub
Backend API feature request
Dominant language
R
Stars
1k
Forks
92
PR merge metrics
No merged PRs in 30d

Description

I have recently discovered [ClusterMQ](https://github.com/mschubert/clustermq) which can run R code in SLURM/LSF/etc. jobs. The biggest advantage over batchtools is it uses ZMQ to transfer data directly to the distributed jobs. In my experience the most serious bottleneck in batchtools is using shared file system (NFS) for data transfer - especially if the data is large.

Contributor guide

Open the contributing guide

Research direction

The issue proposes adding ClusterMQ as a new backend for the future R package, but names no files, tests, or implementation entry points. Start by reviewing the existing backend architecture and ClusterMQ's job and data-transfer model. Done would mean a defined integration scope, implementation, and tests showing that the backend can run distributed jobs.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems, hpc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.