Zostoga still causing memory spikes
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8
- Forks
- 7
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 23
Description
I've tested zostoga with one month of data after [this fix](https://github.com/MetOffice/CDDS/pull/1008) and it seems we're still getting OOM kill events (despite onm mem limit currently being set to 100GB) so this can't be produced for the time being.
I added a bit of profiling to see where the memory is spiking:
```
Loading data for "CMIP7_ocean.json: zostoga_tavg-u-hm-sea"
[memory profile] calc_zostoga: start: RSS=36737.9 MB
[memory profile] calc_zostoga: before processing time step 0: RSS=63702.0 MB
[memory profile] calc_rho_mean: start: RSS=63702.0 MB
[memory profile] eos_insitu: start: RSS=63702.0 MB
[memory profile] eos_insitu: after cast to float64: RSS=93769.3 MB
```
I did do a quick and dirty investigation using copilot to see if there were any quick wins. Unfortunately there doesn't seem to be. It has suggested breaking things up a bit by looping over depth levels in `calc_rho_mean` (and some other bits), which should in theory limit the peak memory usage to ~1 levels worth of data on each loop. This solution looks like it could be quite a complex code change though so i'm not going any further. If anyone is curious as to what that suggested change might look like ( i've left the memory profiling statements in there) , you can have a look [here](https://github.com/MetOffice/CDDS/commit/6be5ea042febc36c4f29eee19b67c6482103644f#diff-e1e33c2f3c6c6f18a3ec708af6113636633feed6bf0a77c61654734087a95554).
Edit: I've tried running with that fix and it does produce with a maximum memory usage of ~68GB. However, i'm lacking the expertise to know if there may be a simpler solution.
Putting this on hold for the time being. Will try to fix in CDDS v4.0.1
@dcalve any thoughts on potential solutions would be appreciated!
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked fix and the profiling commit, then inspect calc_zostoga, calc_rho_mean, and eos_insitu while reproducing the one-month zostoga run. Compare peak memory and output correctness; done means the dataset produces successfully without OOM kills and without changing the calculated results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100