DiamondLightSource / DiamondLightSource/httomo
Handle numpy `_ArrayMemoryError` in reslice `np.concatenate()` call to avoid deadlock in multi-process runs
- Dominant language
- Python
- Stars
- 10
- Forks
- 5
- Avg merge
- 5d 17h
- Merged PRs (30d)
- 1
Description
For single process runs there is no issue, but with multi-process runs with MPI a deadlock can occur if one or more but not all processes run into an `_ArrayMemoryError` at this line in the reslice: https://github.com/DiamondLightSource/httomo/blob/8ccf66304dfb7af0a77495e4bcf974c92f324953/httomo/data/mpiutil.py#L67
The output looks like the following, where some printing has been inserted to show the rank of the processes that run into the error:
```
Pipeline has been separated into 3 sections
See the full log file at: /scratch/yousef/outdata/17-07-2025_15_56_54_output/user.log
Running loader (pattern=projection): standard_tomo...
Finished loader: standard_tomo (httomo) Took 10000.46ms
Section 0 (pattern=projection) with the following methods:
data_reducer (httomolib)
find_center_vo (httomolibgpu)
remove_outlier (httomolibgpu)
normalize (httomolibgpu)
0%| | 0/4 [00:26The center of rotation is 1275.0
Finished processing last block
Rank 0 got an error: Unable to allocate 20.6 GiB for an array with shape (5529600000,) and data type float32
```
Note how only one of the two ranks report the error, and the fact that control is not given back to the terminal after this printing, the terminal is "hanging" and Ctrl-C needs to be done to kill both MPI processes. The specific rank running into the error can vary across different runs, but with a specific machine (`ws582`) and specific test data this deadlock can be consistently reproduced running httomo with 2 MPI processes.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.