DOI-USGS / DOI-USGS/lake-temperature-process-models

When scale up to full footprint, move to writing meteo data to feathers by tile

Open
#42 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
R
Stars
2
Forks
4
PR merge metrics
No merged PRs in 30d

Description

At the moment we're only working in Minnesota, and we are [pulling from the netCDF](https://github.com/USGS-R/lake-temperature-process-models/blob/main/1_prep/src/munge_netCDFs.R#L52-L107) by cell and by time period (opening each of the 6 netCDFs one time each), adding the burn-in and burn out and then writing to .csv. Upon model execution, the correct .csv is copied into each simulation directory.

When we scale up to the full footprint, we should switch to an approach where we instead pull out the data by tile and by time period, storing it in feather files. That way we'd generate (n_tiles*3) files, instead of (n_cells*3) files. That approach will mean that within the execution code, instead of [copying the correct file for each simulation to the simulation directory](https://github.com/USGS-R/lake-temperature-process-models/blob/main/2_run/src/run_glm3.R#L63), we'll need to need to read in the feather file for the correct tile and time period, filter it to the correct cell, and then write the filtered data to the simulation directory. The DS team does not anticipate any issues with multiple `targets` branches simultaneously accessing the same feather file.

Note that the nitty gritty of our approach may be influenced by #41

Contributor guide

Open the contributing guide

Research direction

Start with 1_prep/src/munge_netCDFs.R lines 52-107 and 2_run/src/run_glm3.R line 63 to trace the current netCDF-to-CSV preparation and per-simulation copy. Review issue #41 before settling the approach. Done means preparation uses tile/time-period Feather files and execution filters the relevant tile data for each cell and time period.

Written by the indexing model from the issue text.

Assessment

Tech stack
r
Domain
data-engineering
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.