DiamondLightSource / DiamondLightSource/httomo
Function determining dataset store backing incorrectly assumes new numpy array is always created by reader
- Dominant language
- Python
- Stars
- 10
- Forks
- 5
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 1
Description
In the case of a section that is not the last section, a reader for section `n+1` will be created from the writer used in section `n`. The creation of a reader only causes a new numpy array to be created if there is non-zero padding required for section `n+1`: https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/data/dataset_store.py#L301-L303
https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/data/dataset_store.py#L514-L518
https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/data/dataset_store.py#L503-L506
However, the `determine_store_backing()` function assumes that the creation of the reader will always create a new numpy array, so it accounts for the size of the numpy array even though it'll only exist in the case of non-zero padding: https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/runner/dataset_store_backing.py#L174-L178
## Extra info
For some more info on why I think that there's no new numpy array created for the reader of section `n+1` when there is zero padding: when that case occurs, the reader's `self._data` attribute is assigned to the writer's `self._data` attribute (which is a numpy array) and nothing else will happen to the reader's `self._data`: https://github.com/DiamondLightSource/httomo/blob/f9bbccb3e915f4edfa51f77d5ab6fc195e4d5592/httomo/data/dataset_store.py#L284
Meaning, I think that in the case of zero padding, the reader of section `n+1` simply gets a reference to the numpy array from the writer of section `n` and nothing else (ie, no copy is made, no new array is created) so there's no reason for more memory to be allocated when creating the reader for section `n+1`.
Note that the above info is excluding the case of a reslice: in the case of a reslice, stuff will of course happen in the reslice algorithm to cause allocations, but that is separate from purely what the writer and reader are doing with the numpy arrays that represent the chunks associated with a section.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.