Data is realised during loading, causing very slow loading for large dataset or cutouts
- Dominant language
- Python
- Stars
- 33
- Forks
- 19
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 30
Description
### Describe the bug
At some point during the loading process inside `read.read_cubes` we realise the data. This causes the load to be really slow, especially has this is done before filtering or cutout, so we are loading a lot of data we immediately discard.
In addition to runtime, this might also be affecting job memory usage, which is a big problem for aggregation usage.
### How to reproduce
Steps to reproduce the behaviour:
1. Load some data with a cutout.
2. Observe via system monitor or `/usr/bin/time` that it does far more IO than it should.
### Expected behaviour
Data should not be realised during loading, and even if it was it shouldn't load any more that is needed for the cutout.
Loading data without doing much processing should be fast.
There is a larger piece of work to go through and make pretty much the entirety of CSET lazy, though that should be a separate issue.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.