Possibly excessive memory usage when saving an F03 ancillary file
- Dominant language
- Python
- Stars
- 7
- Forks
- 10
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 5
Description
# 🐜 Bug Report
**This was observed in an earlier version of ANTS** - step 1 should be verifying the problem still exists before any attempted fix.
## How to reproduce
Step by step guide to reproduce the behaviour:
1. Prepare a large netCDF file with one netCDF chunk per field (i.e. per time/level)
2. Save a large cube as an F03 ancillary file (e.g. CMIP6 ozone)
3. Examine memory usage in scheduler job report
Expected behaviour is that one field at a time should be realised. Given delays with garbage collection etc, this probably means a small number of fields, rather than one.
Instead, all fields are retained in memory, and total memory usage is slightly over the output file size.
## Version
This bug exists in ANTS 1.0
## Additional Context
Previous attempted fix was to wrap a `copy.deepcopy()` around `self.ppfield.data` [here](https://github.com/MetOffice/ANTS/blob/dbda2a48738af462776ad85da7ea308a6168a703/lib/ants/fileformats/ancil/__init__.py#L89). This was insufficient to fix the issue in real world testing.
Contributor guide
Research direction
Start by reproducing the report with a large netCDF input containing one chunk per field and saving a large cube as an F03 ancillary file. Inspect lib/ants/fileformats/ancil/__init__.py around the referenced self.ppfield.data line, and compare memory usage with and without the earlier copy.deepcopy() attempt. Done means memory retains only a small number of fields rather than all fields during output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100