[Pre-pull request] Adding jobs to an existing registry
Nobody has claimed this yet.
- Dominant language
- R
- Stars
- 184
- Forks
- 53
- Avg merge
- 7d 2h
- Merged PRs (30d)
- 1
Description
Frequently I run into the situation that I run set of jobs in parallel with batchMap only to realize that I forgot to include an interesting case in the input list at a later date. Historically I've either made a new registry (ugh) or deleted and re-run everything (more-ugh). Today I really didn't want to do either so I figured out how to add jobs to an existing registry.
Is this something you'd consider adding if I put together a PR? I suspect the answer is probably "you should be using the experiment abstraction", but I suspect enough people run in to this problem that it would be beneficial to add. I've included code at the bottom for doing it very roughly, in the case it's going to be a part of the package I'd write something like batchUpdateMap which assembles the new param list for the user.
Example code for doing it manually below for if anyone needs it in the mean-time:
reg <- loadRegistry(reg, writeable = TRUE)
previous_max_id <- max(reg$status$job.id)
new_id <- previous_max_id + 1
new_params <- list(some = pars) #get skeleton from reg$defs$job.pars[[1]]
#Add row to job definitions
reg$defs <-
rbind(reg$defs
, data.table(def.id = new_id
, job.pars = list(list(new_params)))
setkey(reg$defs, "def.id") #reset data.table key
#Add row to status table
reg$status <-
rbind(reg$status
, data.table(job.id = new_id, def.id = new_id, submitted = NA_real_,
started = NA_real_, done = NA_real_, error = NA_character_,
mem.used = NA_real_, resource.id = NA_integer_, batch.id = NA_character_,
log.file = NA_character_, job.hash = NA_character_, job.name = NA_character_,
key = "job.id"))
setkey(reg$status, "job.id") #reset data.table key
saveRegistry(reg) #Save our updates
Obviously this can be generalized for adding more than one job.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the registry operations named in the issue: loadRegistry(), batchMap(), and saveRegistry(), then inspect how reg$defs and reg$status are structured. Done should mean users can add one or more jobs to an existing registry without creating a new registry or rerunning existing jobs, while preserving the registry's definitions and status tables.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100