DiamondLightSource / DiamondLightSource/httomo

Function determining dataset store backing incorrectly assumes new numpy array is always created by reader

Open
#693 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10
Forks
5
Avg merge
5d 17h
Merged PRs (30d)
1

Description

In the case of a section that is not the last section, a reader for section `n+1` will be created from the writer used in section `n`. The creation of a reader only causes a new numpy array to be created if there is non-zero padding required for section `n+1`: https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/data/dataset_store.py#L301-L303

https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/data/dataset_store.py#L514-L518

https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/data/dataset_store.py#L503-L506

However, the `determine_store_backing()` function assumes that the creation of the reader will always create a new numpy array, so it accounts for the size of the numpy array even though it'll only exist in the case of non-zero padding: https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/runner/dataset_store_backing.py#L174-L178

## Extra info

For some more info on why I think that there's no new numpy array created for the reader of section `n+1` when there is zero padding: when that case occurs, the reader's `self._data` attribute is assigned to the writer's `self._data` attribute (which is a numpy array) and nothing else will happen to the reader's `self._data`: https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/data/dataset_store.py#L284

Meaning, I think that in the case of zero padding, the reader of section `n+1` simply gets a reference to the numpy array from the writer of section `n` and nothing else (ie, no copy is made, no new array is created) so there's no reason for more memory to be allocated when creating the reader for section `n+1`.

Note that the above info is excluding the case of a reslice: in the case of a reslice, stuff will of course happen in the reslice algorithm to cause allocations, but that is separate from purely what the writer and reader are doing with the numpy arrays that represent the chunks associated with a section.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.