futureverse / futureverse/future
BACKEND: ClusterMQ as a new backend
- Dominant language
- R
- Stars
- 1k
- Forks
- 92
- PR merge metrics
- No merged PRs in 30d
Description
I have recently discovered [ClusterMQ](https://github.com/mschubert/clustermq) which can run R code in SLURM/LSF/etc. jobs. The biggest advantage over batchtools is it uses ZMQ to transfer data directly to the distributed jobs. In my experience the most serious bottleneck in batchtools is using shared file system (NFS) for data transfer - especially if the data is large.
Contributor guide
Research direction
The issue proposes adding ClusterMQ as a new backend for the future R package, but names no files, tests, or implementation entry points. Start by reviewing the existing backend architecture and ClusterMQ's job and data-transfer model. Done would mean a defined integration scope, implementation, and tests showing that the backend can run distributed jobs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- distributed-systems, hpc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100