DiamondLightSource / DiamondLightSource/httomo

Handle numpy `_ArrayMemoryError` in reslice `np.concatenate()` call to avoid deadlock in multi-process runs

Open
#602 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
10
Forks
5
Avg merge
5d 17h
Merged PRs (30d)
1

Description

For single process runs there is no issue, but with multi-process runs with MPI a deadlock can occur if one or more but not all processes run into an `_ArrayMemoryError` at this line in the reslice: https://github.com/DiamondLightSource/httomo/blob/8ccf66304dfb7af0a77495e4bcf974c92f324953/httomo/data/mpiutil.py#L67

The output looks like the following, where some printing has been inserted to show the rank of the processes that run into the error:
```
Pipeline has been separated into 3 sections
See the full log file at: /scratch/yousef/outdata/17-07-2025_15_56_54_output/user.log
Running loader (pattern=projection): standard_tomo...
Finished loader: standard_tomo (httomo) Took 10000.46ms
Section 0 (pattern=projection) with the following methods:
data_reducer (httomolib)
find_center_vo (httomolibgpu)
remove_outlier (httomolibgpu)
normalize (httomolibgpu)
0%| | 0/4 [00:26The center of rotation is 1275.0
Finished processing last block
Rank 0 got an error: Unable to allocate 20.6 GiB for an array with shape (5529600000,) and data type float32
```

Note how only one of the two ranks report the error, and the fact that control is not given back to the terminal after this printing, the terminal is "hanging" and Ctrl-C needs to be done to kill both MPI processes. The specific rank running into the error can vary across different runs, but with a specific machine (`ws582`) and specific test data this deadlock can be consistently reproduced running httomo with 2 MPI processes.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.