DOI-USGS / DOI-USGS/scipiper

Reduce number of .yml reads in loop_tasks

Open
#77 4 comments 0 reactions 3 assignees Claimed by @wdwatkins View on GitHub
Dominant language
R
Stars
10
Forks
12
PR merge metrics
No merged PRs in 30d

Description

`loop_tasks()` can be quite slow for large task makefiles. A current hypothesis is that a major cause of this slowness is that `scmake` gets (called separately for every target)[https://github.com/USGS-R/scipiper/blob/e5e90b11200aaf4448683889c7bd0ea487b6e9dd/R/loop_tasks.R#L138], and that this is slow because `scmake` calls `remake` rereads the task .yml file and rebuilds the dependency tree every time.

For this task, find a way to only read the task .yml file once within that loop over `incomplete_targets`. This might involve commands that dig into the remake code a bit, perhaps things like
```
remake_object <- remake:::remake(remake_file=remake_file, verbose=FALSE, load_sources=TRUE)
```
which could be called just once and then called to some other remake function that does the making based on that remake_object. Look at the source code for `remake::make` to figure out what is needed and which internal functions might be called.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.