gotchya! including `remake.yml` can really slow down complex pipeline task tables
- Dominant language
- R
- Stars
- 10
- Forks
- 12
- PR merge metrics
- No merged PRs in 30d
Description
**The issue:** I have been working in https://github.com/USGS-R/lake-temperature-out and running the `3_summarize` part of the pipeline. So far, I have had no issue building task tables with 880 tasks that take ~8 min each in parallel on Yeti UV. I just started to build this for a task table with about 7,000 tasks and have struggled. It gets hung up prior to starting the loop, as it checked which targets in the task table needed to be rebuilt. One of the earlier suggestions (from @aappling-usgs) to fix this issue was to save indicator files rather than use objects or actual data files as targets in the task table. I implemented this approach, but without any noticeable improvements. Even trying to build just one target in this big task makefile (a la `scmake("task_target", "task_makefile.yml")`) was impossible because it would take hours before it would start the actual build (I didn't wait to see how long because 1 hour to figure out how to build a target that should take 10 minutes seemed ridiculous).
**The current solution:** After talking with @jread-usgs, it became clear that listing the `remake.yml` in the `include:` field of the task makefile might be causing this slow down. When this is added, it is adding complexity to an already lengthy dependency tree. I was using `include: remake.yml` because there were a few object targets that were being used in the task steps. The current solution is to decouple the task makefile from the rest of the pipeline by removing `remake.yml` from `include:` and then saving the required object targets as temporary files during the task makefile creation (which get deleted once `loop_tasks` is complete). I am still testing this out right now, but it worked well when I built just one task in the huge task makefile. The single task took 8 minutes, which is what I expected (the same attempt before took longer than an hour ...). When I kicked off the full build, it took 25 minutes to get from kicking off the build to printing `### Starting loop attempt ... `. Way better than the hours I had waited previously ... sometimes my Yeti UV job would just die without any errors.
**Some concerns:** Since we are building these tasks in parallel (I am currently using 80 cores on Yeti UV), will it be an issue that we have 80 functions trying to all read in the same RDS file at once? We aren't sure yet.
**Possible outcomes:**
1. Should we consider other code options? One idea I had was that it would be nice to add some targets to the task makefile that aren't repeated per task. This would have allowed me to add the object targets one time at the beginning of the task makefile and then repeat the rest. We could do this in a separate temporary makefile, too. These options seemed more convoluted and required more effort than switching the object targets to files.
2. Does this "gotchya" need to be documented somewhere in `scipiper`? Would a user get notified if this is a concern with how they have setup their pipeline? Perhaps some threshold of number of targets in the task table could be used to push a warning?
3. Does the approach need to be outlined in a future pipelines training: "considerations when scaling up task tables"?
Contributor guide
Research direction
Read the remake.yml inclusion path and the scmake(), task_makefile.yml, and loop_tasks entry points described in the report. Reproduce the large task-table delay with and without including remake.yml, then determine whether the accepted outcome is documentation, a warning, or a task-makefile design change; the issue does not define a single completion criterion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- build-system, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100