DOI-USGS / DOI-USGS/scipiper

not all indicator files should be YAMLified

Open
#10 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
10
Forks
12
PR merge metrics
No merged PRs in 30d

Description

I think there are times when we won't want to commit some indicator files that are referenced only within a task makefile. This would allow us to avoid having to commit hundreds of task-level indicator files plus their corresponding build status files (assuming a shared-cache system). This would be appropriate when we are confident that only one person will need to run the tasks within that job, or that if another person needs to access intermediate files from the job, it won't be excessively onerous to rerun the entire job.

## Example
In `watersat` project, the main `remake.yml` creates and loops through tasks in `tasks_1_wqp.ind`. The looping has intermediate products of `partition_Alaska_secchi_001` -> `1_wqdata/tmp/wqp/Alaska_secchi_001.feather.ind` -> `1_wqdata/out/wqp/Alaska_secchi_001.feather.ind` -> `1_wqdata/out/wqp/Alaska_secchi_001.feather` (and so on for other states and constituents). The looping has a final product of `1_wqdata/log/tasks_1_wqp.ind`.
* `1_wqdata/log/tasks_1_wqp.ind` should always be YAMLified and shared to make the shared cache work, because it's referenced in the main `remake.yml` and downstream files depend on it.
* Option 1 for the indicator files referenced only within `tasks_1_wqp.yml`: If the `out/*.ind` files are YAMLified and committed, then the `tmp/*.ind` files should also be YAMLified and committed, because otherwise remake will think the `out/*.ind` files are out of date. This is the approach I'm choosing for this particular project and job. `scmake` has automatically prepared build status files for all of these .ind files, so those should also be committed.
* Option 2 for the indicator files referenced only within `tasks_1_wqp.yml`: We could choose not to commit any of these indicator files, as suggested at the top of this issue. This would be nice because we could avoid committing several hundred indicator and build status files. This would be a better option if our tasks didn't involve pushing files to the shared cache, which we do in this example, because to share files in the cache without sharing the indicator files means nobody but the originator can access the shared-cache files properly via scipiper; everyone else would have to go outside the dependency manager or redo the entire job despite the fact that the files are already shared.

## Implementation
To implement option 2 from the example (i.e., to not share some indicator files), I think we would give those unshared indicator files a different extension.
* `scmake` calls `YAMLify_build_status` only on file targets that pass `is_indicator`, which looks to `indicator_extension()` for a single indicator extension. This part should work right away
* `create_task_plan`'s `add_complete` option is a simple T/F logical which adds an `sc_indicate` command if TRUE. Therefore we would either need to (1) use `add_complete=FALSE`, possibly writing our own indicator file with a different indicator extension, or (2) modify the `create_task_plan` so that it's easier to request a non-standard indicator file extension for task completeness indicators.
* consider making shared and unshared indicator file extensions both be first-class citizens of scipiper.
This would mean modifying functions like `sc_indicate`, `create_task_plan`, and maybe also `gd_put` and `s3_put` so that they can all write a specified indicator type. `indicator_extension` would need to permit two extensions. and so on.
* then again, maybe unshared indicator files should continue as second-class citizens. we could modify `create_task_plan` to be more flexible (as above) and could either make `sc_indicate` more flexible or create a new `uc_indicate` (where 'uc'='unshared cache') for these special cases.

Contributor guide

Open the contributing guide

Research direction

Start by reading scmake, indicator_extension, create_task_plan, and sc_indicate to understand how indicator and build-status files are selected and created. Review the watersat example and the mentioned gd_put and s3_put paths. Done means a documented, tested way to keep selected task-level indicators unshared without breaking shared-cache dependencies.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
build-system, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.