DOI-USGS / DOI-USGS/scipiper

feature request - fault tolerant builds for loop_tasks when the files all exist

Open
#124 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
R
Stars
10
Forks
12
PR merge metrics
No merged PRs in 30d

Description

No immediate priority on this because I think we all have a way around this (e.g., deleting all of the files/targets).

Sometimes the tasks we are using `loop_tasks` for are brittle (e.g., downloads on shaky services). If a rebuild is triggered because some params or code changes and you already have files for the tasks, the fault tolerant part of `loop_tasks` is skipped over and you go right to `scmake`, which will only try once and error on the first fail. Ideally I think we'd still get the fault tolerant part of this function if the targets already exist but they are stale.

Contributor guide

Open the contributing guide

Research direction

Start by tracing the `loop_tasks` implementation and the point where it hands existing or stale targets to `scmake`. Reproduce a rebuild with all task files present and an intermittent download failure, then verify that the fault-tolerant retry behavior is still used and the build completes when a retry succeeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
build-system
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.