DOI-USGS / DOI-USGS/lake-temperature-process-models
When scale up to full footprint, move to writing meteo data to feathers by tile
- Dominant language
- R
- Stars
- 2
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
At the moment we're only working in Minnesota, and we are [pulling from the netCDF](https://github.com/USGS-R/lake-temperature-process-models/blob/main/1_prep/src/munge_netCDFs.R#L52-L107) by cell and by time period (opening each of the 6 netCDFs one time each), adding the burn-in and burn out and then writing to .csv. Upon model execution, the correct .csv is copied into each simulation directory.
When we scale up to the full footprint, we should switch to an approach where we instead pull out the data by tile and by time period, storing it in feather files. That way we'd generate (n_tiles*3) files, instead of (n_cells*3) files. That approach will mean that within the execution code, instead of [copying the correct file for each simulation to the simulation directory](https://github.com/USGS-R/lake-temperature-process-models/blob/main/2_run/src/run_glm3.R#L63), we'll need to need to read in the feather file for the correct tile and time period, filter it to the correct cell, and then write the filtered data to the simulation directory. The DS team does not anticipate any issues with multiple `targets` branches simultaneously accessing the same feather file.
Note that the nitty gritty of our approach may be influenced by #41
Contributor guide
Research direction
Start with 1_prep/src/munge_netCDFs.R lines 52-107 and 2_run/src/run_glm3.R line 63 to trace the current netCDF-to-CSV preparation and per-simulation copy. Review issue #41 before settling the approach. Done means preparation uses tile/time-period Feather files and execution filters the relevant tile data for each cell and time period.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- data-engineering
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100