DOI-USGS / DOI-USGS/scipiper

improve efficiency of loop_tasks is_current check

Open
#65 3 comments 0 reactions 3 assignees Claimed by @wdwatkins View on GitHub
Dominant language
R
Stars
10
Forks
12
PR merge metrics
No merged PRs in 30d

Description

I was just looking at https://github.com/USGS-R/scipiper/blob/master/R/loop_tasks.R#L70-L76 again and am not sure it's as efficient as I initially thought. In particular, we call `get_remake_status(targets, task_makefile)` on _all_ possible targets and only later exclude those that already have an existing file. But the check for whether existing files are current is the slow part, so we surely shouldn't be calling `get_remake_status(targets, task_makefile)` on those existing files at all, right?

And maybe we should make it a user option to either loop over all tasks or to loop over complete unbuilt tasks (for which file doesn't yet exist)? Sometimes the hashing is expensive while the retries of unbuilt files are the focus; other times the hashing is cheap and we'd like to include retries for built but out-of-date files (e.g., when we forget or don't know to or don't want to delete the files before building).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.