mlr-org / mlr-org/batchtools

Creating simple experiment from function without "data" argument

Open
#132 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
R
Stars
184
Forks
53
Avg merge
7d 2h
Merged PRs (30d)
1

Description

I am struggling to design an experiment using my own function, which I will try to illustrate here using a function piApprox() from Example 1 from "Get Started":

piApprox = function(n) {
  nums = matrix(runif(2 * n), ncol = 2)
  d = sqrt(nums[, 1]^2 + nums[, 2]^2)
  4 * mean(d <= 1)
}

I now want to create and run 50 jobs using piApprox() in a way they will satisfy 3 conditions:

  1. I want to run piApprox() with 5 different values of parameter n: n1=1000, n2=2000, n3=3000, n4=4000, n5=5000, with 10 jobs created for each value of n

  2. Each of these 5 parametrizations to start from seed=1 so these 5 sets of results stay stochastically comparable

  3. For convenience, I want to keep all the code for definition/configuration of this example as single R script, ideally creating only a single registry

My first solution is to run all 5*10 jobs together like this:

reg = makeRegistry(file.dir = NA, seed = 1)
batchMap(fun = piApprox, n = rep(1:5, each=10)*1000)

Upper solution would satisfy conditions 1 and 3, but not the important condition 2 about seeding

My second solution I can think of is to create five separate scripts/registers each with separate value of parameter n, satisfying conditions 1 and 2, but tedious to set up and collect results

Now I believe this is possible to define this assignment using addProblem(), addAlgorithm(), addExperiment() functions in a way the solution would satisfy my 3 aforementioned conditions. But as a beginner, I am struggling to come up with such solution (e.g. what can I pass as the data parameter to addProblem(), when piApprox() is generating its own data?).

I think that an exercise similar to what I am describing could serve as helpful bridging example between simplest Example1 and more advanced Example2.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the batchtools “Get Started” Example 1 and Example 2, then read the roles of addProblem(), addAlgorithm(), and addExperiment(). The example is done when one R script and registry demonstrate five n values, ten jobs per value, comparable seeds, and collected results without requiring a data argument.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
hpc
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.