MetOffice / MetOffice/CDDS

Zostoga still causing memory spikes

Open
#1,012 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

on hold
Dominant language
Python
Stars
8
Forks
7
Avg merge
1d 15h
Merged PRs (30d)
23

Description

I've tested zostoga with one month of data after [this fix](https://github.com/MetOffice/CDDS/pull/1008) and it seems we're still getting OOM kill events (despite onm mem limit currently being set to 100GB) so this can't be produced for the time being.

I added a bit of profiling to see where the memory is spiking:
```
Loading data for "CMIP7_ocean.json: zostoga_tavg-u-hm-sea"
[memory profile] calc_zostoga: start: RSS=36737.9 MB
[memory profile] calc_zostoga: before processing time step 0: RSS=63702.0 MB
[memory profile] calc_rho_mean: start: RSS=63702.0 MB
[memory profile] eos_insitu: start: RSS=63702.0 MB
[memory profile] eos_insitu: after cast to float64: RSS=93769.3 MB
```
I did do a quick and dirty investigation using copilot to see if there were any quick wins. Unfortunately there doesn't seem to be. It has suggested breaking things up a bit by looping over depth levels in `calc_rho_mean` (and some other bits), which should in theory limit the peak memory usage to ~1 levels worth of data on each loop. This solution looks like it could be quite a complex code change though so i'm not going any further. If anyone is curious as to what that suggested change might look like ( i've left the memory profiling statements in there) , you can have a look [here](https://github.com/MetOffice/CDDS/commit/6be5ea042febc36c4f29eee19b67c6482103644f#diff-e1e33c2f3c6c6f18a3ec708af6113636633feed6bf0a77c61654734087a95554).

Edit: I've tried running with that fix and it does produce with a maximum memory usage of ~68GB. However, i'm lacking the expertise to know if there may be a simpler solution.

Putting this on hold for the time being. Will try to fix in CDDS v4.0.1

@dcalve any thoughts on potential solutions would be appreciated!

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked fix and the profiling commit, then inspect calc_zostoga, calc_rho_mean, and eos_insitu while reproducing the one-month zostoga run. Compare peak memory and output correctness; done means the dataset produces successfully without OOM kills and without changing the calculated results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.