DiamondLightSource / DiamondLightSource/httomo

Inconsistent creation of blocks by test loader compared to real loader

Open
#579 0 comments 0 reactions 0 assignees View on GitHub
testing
Dominant language
Python
Stars
10
Forks
5
Avg merge
5d 17h
Merged PRs (30d)
1

Description

The real loader creates a new numpy array (into which a subset of the input data in the hdf5 file is copied): https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/loaders/standard_tomo_loader.py#L171

which then eventually gets put into a `DataSetBlock`: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/loaders/standard_tomo_loader.py#L304-L313

The test loader defined in `testing_utils.py` which can be configured to produce one or more mock blocks:
- takes in a `DataSetBlock`
- and then uses slices of the numpy array within that `DataSetBlock`

to create new blocks: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/tests/testing_utils.py#L86-L98

Ie, if running httomo within a test that wants multiple blocks to be read from the test loader, the loader is providing multiple blocks containing references to a single numpy array. But if running httomo not in a test, the real loader provides multiple blocks where each block has its own data, rather than referencing a single central numpy array.

This can cause different behaviour between tests and real runs if more than one block is in a section due to the `del block.data` line in the task runner: https://github.com/DiamondLightSource/httomo/blob/4e4b9d9bdf6e69d97aff840554ec9630b048e2c4/httomo/runner/task_runner.py#L158-L161

For the case of multiple blocks in a test produced by the test loader, the reference count of the single central numpy array drops to 0 due to the `del block.data`, and subsequent attempts to read blocks fails due to the single central numpy array not being there anymore:
```
File "/httomo/tests/runner/test_task_runner.py", line 735, in test_minimum_block_length_passed_to_intermediate_data_wrapper
t._execute_section(s[0])
File "/httomo/httomo/runner/task_runner.py", line 160, in _execute_section
del block.data
^^^^^^^^^^
File "/opt/conda/lib/python3.12/unittest/mock.py", line 823, in __delattr__
raise AttributeError(name)
AttributeError: data
```

The test loader should be modified to:
- take a plain numpy array rather than a `DataSetBlock`
- copy data from that numpy array into a new numpy, before putting the new numpy array into a new `DataSetBlock`

in order to be consistent with how a real loader works when multiple blocks are loaded from it.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.