rush adapter for mlr3 benchmarks?

hi @be-marc
In machine learning experiments, some algorithms can take a lot of time (deep neural networks), whereas others can be very fast (featureless baseline or linear model). Also there can be different data sets, with very different sizes in terms of numbers of rows and columns. For example Table 1 of our SOAK paper shows 20 data sets, with ~1,000 to ~1,000,000 rows, and 10 to ~10,000 features (1000x differences). So it does not really make sense to schedule these different algorithms and data sets with the same time limits (as is required by SLURM job arrays).

It would be great if there was some easy way to run mlr3 benchmarks on a rush network, something like mlr3batchmark --- mlr3rushmark?

Here is my desired workflow:

• create benchmark_grid in central R process.
• run mlr3rushmark::rushmark() to convert the benchmark_grid to a batchtools job table (different rows for different combinations of algos/tasks/resampling iterations).
• run batchtools::submitJobs() to start the cluster jobs.
• maybe the first cluster job starts a redis database, one key for each batchtools job table row?
• each cluster job is running a rush worker which looks at the redis database to determine a new job to run.
• at the end it would be nice to have some function mlr3rushmark::reduceResultsRushmark(), something like mlr3batchmark::reduceResultsBatchmark().

do you think that would be possible?

For the problem of scheduling machine learning benchmarks on a HPC cluster, I think this would be a better solution than the "arrays" proposition we discussed last week https://github.com/mlr-org/mlr3batchmark/issues/32#issuecomment-2883302970

• arrays proposition runs several benchmark iterations in the same cluster worker, but systematically (say all algos for a given task and resampling iteration, fixed number of benchmark iterations), so still suffers from job heterogeneity (some cluster jobs fast for smaller tasks, others slow for large tasks, not easy to ask the cluster for the right amount of time for each job). Result is some cluster workers terminate early, others much later.
• mlr3rushmark proposition keeps running benchmark iterations in all allocated cluster jobs, until there is no more work to do, which results in all cluster workers terminating at approximately the same time (after computing a variable number of benchmark iterations).

Start by reviewing the existing mlr3batchmark workflow and the proposed rush worker and Redis coordination model described in the issue. Assess how benchmark_grid conversion, batchtools::submitJobs(), rush workers, and result reduction would fit together; done should include a validated workflow or a clearly scoped implementation plan for these functions.