MetOffice / MetOffice/ANTS

defer_cube: Investigate replacing cube saving with data payload saving

Open
#145 3 comments 0 reactions 1 assignee Claimed by @arjclark View on GitHub
➡ medium priority 🐜 bug
Dominant language
Python
Stars
7
Forks
10
Avg merge
1d 19h
Merged PRs (30d)
5

Description

# ✨ Feature Request
In the `cube.defer_cube()` routine we save cubes to disk and then load them back in so that the data payloads are lazy. We should only save the data and load that back into the cube.

## Requirements

No significant changes in resource overheads for swapping from saving/loading cubes to cube.data payload saving either via the dask or numpy method as appropriate.

## Context

Iris does not guarantee round tripping of save > load loops - items can change/get reprocessed on load meaning what was saved is not necessarily what you get back. In the decomposition this is unhelpful as what you get out the end should be the same as what you would get if you hadn't used it at all. In practice, given `cube.defer_cube()` is used in the decomposition with splitting up and recombining cubes, we should replace the iris cube operations with ones that only work with the data. In practice this should be a minor change in terms of number of lines changed. The change should be tested in anger by one of the workflow owners to make sure there are no unexpected resource changes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.