DOI-USGS / DOI-USGS/lake-temperature-process-models
Set up GLM code to use "DataWarp" burst buffer for GLM I/O operations
- Dominant language
- R
- Stars
- 2
- Forks
- 4
- PR merge metrics
- No merged PRs in 30d
Description
### **Note: I'm pulling this discussion (previously in #2) out into a new issue, since it's now a defined task, but this is currently _TABLED_ until we progress further on other higher priority tasks for the GLM modeling work**
_____________________________________________________________
During the last [full run of all 64,494 models](https://github.com/USGS-R/lake-temperature-process-models/issues/2#issuecomment-1033243751), I had vmstat tracking going in order to track memory usage, CPU usage, and I/O demands.
Jesse dug into those results, and is pretty certain that the performance bottleneck is I/O. The system is never coming anywhere near running out of RAM, and the CPU is only ever used ~30% at most:

David [suggested](https://github.com/USGS-R/lake-temperature-process-models/issues/2#issuecomment-1023672082) looking into writing GLM output to a scratch directory, so that "each model was writing to an independent file system and not competing with each other on /cxfs". Jesse explored options for Denali and [identified](https://github.com/USGS-R/lake-temperature-process-models/issues/2#issuecomment-1027174397) the "DataWarp" [burst buffer](https://hpcportal.cr.usgs.gov/hpc-user-docs/Denali/DataWarp.html) as the best supported option. He's thinking the process would involve:
1. Estimating the total highs-speed scratch space storage needs,
2. Getting comfortable with requesting the necessary DataWarp resources through slurm and accessing them in the script, and
2. Doing the actual changes to the pipeline to make it use this resource. It looks to me like we'd be using what they describe as "scratch" here, so the necessary changes would be (a) Setting GLM's input to use a DataWarp scratch directory, which would be named by the environment variable `Sys.getenv("DW_JOB_STRIPED")`, and (b) setting the outputs to go first to a scratch directory in `Sys.getenv("DW_JOB_STRIPED")`, then be moved to the regular targets folder hierarchy and the scratch directory deleted.
I did some [investigation into (1)](https://github.com/USGS-R/lake-temperature-process-models/issues/2#issuecomment-1031999666), and Jesse is thinking that we could ask for ~10 GB of the 46 TB of this space on Denali.
For reference, Jordan also noted
> GLM outputs scale predictably based on number of timesteps in the output (we do one per day) and the "max_layers" param in the .nml, since GLM populates those regardless of whether they are wet or dry. So for every year and every 100 layers, your file grows by ~2.9MB.
I have not yet started work on items (2) or (3)
Contributor guide
Research direction
Start by reading the linked Denali DataWarp documentation and the existing GLM pipeline and Slurm resource requests. Estimate the required scratch space, then trace how GLM inputs and outputs are configured around Sys.getenv("DW_JOB_STRIPED"). Done means the pipeline requests and accesses DataWarp scratch, stages inputs and outputs correctly, and removes temporary data after the run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- r
- Domain
- hpc, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100