mlr-org / mlr-org/batchtools

How to retrieve benchmark results from makeExperimentRegistry()

Open
#249 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
R
Stars
184
Forks
53
Avg merge
7d 2h
Merged PRs (30d)
1

Description

I am trying to use mlr with batchtools to conduct benchmarking. In order to generate a prediction on the benchmarked results from batchtools, I need to retrieve the benchmark results. Using reduceResultsList doesn't work with mlr's generateThreshVsPerfData. Is there another function to retrieve the results? Because results from reduceResultsList() is not acceptable in MLR to generate ROC curve.

#retrieve benchmark result
result = reduceResultsList(ids = c(c(1:284), c(286:318)), reg = regis, missing.val = NA)
Reducing [===================================================================================================>] 100% eta: 0s>

df = generateThreshVsPerfData(result, measures = list(fpr, tpr, mmce))
Error in generateThreshVsPerfData.list(result, measures = list(fpr, tpr, :
Assertion on 'obj' failed: May only contain the following types: Prediction,ResampleResult.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with makeExperimentRegistry() and the reduceResultsList() and generateThreshVsPerfData() calls shown in the issue. Review batchtools and mlr result-handling documentation to identify the compatible benchmark result representation; done means the retrieved results can be passed to generateThreshVsPerfData without the reported type assertion error.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
distributed-systems, hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.