DiamondLightSource / DiamondLightSource/httomo
Inconsistent creation of blocks by test loader compared to real loader
- Dominant language
- Python
- Stars
- 10
- Forks
- 5
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 1
Description
The real loader creates a new numpy array (into which a subset of the input data in the hdf5 file is copied): https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/loaders/standard_tomo_loader.py#L171
which then eventually gets put into a `DataSetBlock`: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/loaders/standard_tomo_loader.py#L304-L313
The test loader defined in `testing_utils.py` which can be configured to produce one or more mock blocks:
- takes in a `DataSetBlock`
- and then uses slices of the numpy array within that `DataSetBlock`
to create new blocks: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/tests/testing_utils.py#L86-L98
Ie, if running httomo within a test that wants multiple blocks to be read from the test loader, the loader is providing multiple blocks containing references to a single numpy array. But if running httomo not in a test, the real loader provides multiple blocks where each block has its own data, rather than referencing a single central numpy array.
This can cause different behaviour between tests and real runs if more than one block is in a section due to the `del block.data` line in the task runner: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/runner/task_runner.py#L158-L161
For the case of multiple blocks in a test produced by the test loader, the reference count of the single central numpy array drops to 0 due to the `del block.data`, and subsequent attempts to read blocks fails due to the single central numpy array not being there anymore:
```
File "/httomo/tests/runner/test_task_runner.py", line 735, in test_minimum_block_length_passed_to_intermediate_data_wrapper
t._execute_section(s[0])
File "/httomo/httomo/runner/task_runner.py", line 160, in _execute_section
del block.data
^^^^^^^^^^
File "/opt/conda/lib/python3.12/unittest/mock.py", line 823, in __delattr__
raise AttributeError(name)
AttributeError: data
```
The test loader should be modified to:
- take a plain numpy array rather than a `DataSetBlock`
- copy data from that numpy array into a new numpy, before putting the new numpy array into a new `DataSetBlock`
in order to be consistent with how a real loader works when multiple blocks are loaded from it.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.