MetOffice / MetOffice/ANTS

Possibly excessive memory usage when saving an F03 ancillary file

Open
#140 0 comments 0 reactions 0 assignees View on GitHub
🐜 bug
Dominant language
Python
Stars
7
Forks
10
Avg merge
1d 19h
Merged PRs (30d)
5

Description

# 🐜 Bug Report

**This was observed in an earlier version of ANTS** - step 1 should be verifying the problem still exists before any attempted fix.

## How to reproduce
Step by step guide to reproduce the behaviour:

1. Prepare a large netCDF file with one netCDF chunk per field (i.e. per time/level)
2. Save a large cube as an F03 ancillary file (e.g. CMIP6 ozone)
3. Examine memory usage in scheduler job report

Expected behaviour is that one field at a time should be realised. Given delays with garbage collection etc, this probably means a small number of fields, rather than one.

Instead, all fields are retained in memory, and total memory usage is slightly over the output file size.

## Version
This bug exists in ANTS 1.0

## Additional Context

Previous attempted fix was to wrap a `copy.deepcopy()` around `self.ppfield.data` [here](https://github.com/MetOffice/ANTS/blob/dbda2a48738af462776ad85da7ea308a6168a703/lib/ants/fileformats/ancil/__init__.py#L89). This was insufficient to fix the issue in real world testing.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the report with a large netCDF input containing one chunk per field and saving a large cube as an F03 ancillary file. Inspect lib/ants/fileformats/ancil/__init__.py around the referenced self.ppfield.data line, and compare memory usage with and without the earlier copy.deepcopy() attempt. Done means memory retains only a small number of fields rather than all fields during output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.