How to retrieve benchmark results from makeExperimentRegistry()
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 184
- Forks
- 53
- Avg merge
- 7d 2h
- Merged PRs (30d)
- 1
Description
I am trying to use mlr with batchtools to conduct benchmarking. In order to generate a prediction on the benchmarked results from batchtools, I need to retrieve the benchmark results. Using reduceResultsList doesn't work with mlr's generateThreshVsPerfData. Is there another function to retrieve the results? Because results from reduceResultsList() is not acceptable in MLR to generate ROC curve.
#retrieve benchmark result
result = reduceResultsList(ids = c(c(1:284), c(286:318)), reg = regis, missing.val = NA)
Reducing [===================================================================================================>] 100% eta: 0s>
df = generateThreshVsPerfData(result, measures = list(fpr, tpr, mmce))
Error in generateThreshVsPerfData.list(result, measures = list(fpr, tpr, :
Assertion on 'obj' failed: May only contain the following types: Prediction,ResampleResult.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with makeExperimentRegistry() and the reduceResultsList() and generateThreshVsPerfData() calls shown in the issue. Review batchtools and mlr result-handling documentation to identify the compatible benchmark result representation; done means the retrieved results can be passed to generateThreshVsPerfData without the reported type assertion error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- distributed-systems, hpc
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100