MetOffice / MetOffice/CSET

Data is realised during loading, causing very slow loading for large dataset or cutouts

Open
#1,794 0 comments 0 reactions 1 assignee Claimed by @jfrost-mo View on GitHub
bug R2O Trials
Dominant language
Python
Stars
33
Forks
19
Avg merge
1d 22h
Merged PRs (30d)
30

Description

### Describe the bug

At some point during the loading process inside `read.read_cubes` we realise the data. This causes the load to be really slow, especially has this is done before filtering or cutout, so we are loading a lot of data we immediately discard.

In addition to runtime, this might also be affecting job memory usage, which is a big problem for aggregation usage.

### How to reproduce

Steps to reproduce the behaviour:

1. Load some data with a cutout.
2. Observe via system monitor or `/usr/bin/time` that it does far more IO than it should.

### Expected behaviour

Data should not be realised during loading, and even if it was it shouldn't load any more that is needed for the cutout.
Loading data without doing much processing should be fast.

There is a larger piece of work to go through and make pretty much the entirety of CSET lazy, though that should be a separate issue.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.