CERN / CERN/TIGRE

TVdenoising invalid argument on multiple GPUs

Open
#657 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
MATLAB
Stars
803
Forks
262
Avg merge
9d 13h
Merged PRs (30d)
8

Description

Hello, I'm running into some memory problems while reconstructing some data using ossart_tv. Normal ossart works fine and I've narrowed down the problem to the tvdenoise function (from _tv_proximal) called from utilities/im_3d_denoise.py. My goal is to reconstruct data into a 2000^3 volume. Other algorithms like fdk, sirt, os_pcsd, and os_asd_pocs works fine.

I use TIGRE version 3.0.0 in Python with the following versions:
- Python/3.10.4
- GCC/11.3.0
- CUDA/12.1.1
- cuDNN/8.9.2.26

Hardware (HPC):
- CPU: Intel Xeon Platinum 8470
- GPU: NVIDIA H100 80GB HBM3

I've tried the code with 1, 2, and 3 GPUs.

A minimal example which replicates im_3d_denoise.py:
```
import numpy as np
from _tv_proximal import tvdenoise
from tigre.utilities.gpu import GpuIds

for i in range(1000, 2100, 100):
print(f'{i = }')

img = np.random.rand(i, i, i).astype(np.float32)
gpuids=GpuIds()
lmbda=50
iter=50

print(f'Image memory consumption: {img.nbytes / 1024 / 1024 / 1024:.4f} GB.')

img = tvdenoise(img, iter, lmbda, gpuids)
```

This code produces the following error for some different combinations of cpu memory and number of GPUs:

**CPU memory 512GB, 1 GPU: Error at i=2000 - Image memory = 30GB**
Common/CUDA/TIGRE_common.cpp (18): tvDenoise:tvdenoising:Memory TV dneoising requires 5 times the image memory. Your GPU(s) do not have the required memory.
This memory will be attempted to allocate on the CPU, Whic may fail or slow the computation by a very significant amount.

**CPU memory 512GB, 2 GPUs: Error at i=1600 - Image memory = 16GB**
Common/CUDA/TIGRE_common.cpp (7): TV minimization
Common/CUDA/TIGRE_common.cpp (14): CBCT:CUDA:TVdenoising invalid argument

**CPU memory 512GB, 3 GPUs: Error at i=1600 - Image memory = 16GB**
Common/CUDA/TIGRE_common.cpp (7): TV minimization
Common/CUDA/TIGRE_common.cpp (14): CBCT:CUDA:TVdenoising invalid argument

**CPU memory 128GB, 3 GPUs: Error at i=1600 - Image memory = 16GB**
Common/CUDA/TIGRE_common.cpp (7): TV minimization
Common/CUDA/TIGRE_common.cpp (14): CBCT:CUDA:TVdenoising invalid argument

From my understanding I would need around 150GB (2000^3) of memory on the GPUs, i.e. 2 or even 3 GPUs should be enough (80GB each).

- Do you know why this happens? Maybe it makes sense, but my profficiency in CUDA code is not the best.
- Do you have any suggestions as to what I can try to fix this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.