TVdenoising invalid argument on multiple GPUs
- Dominant language
- MATLAB
- Stars
- 803
- Forks
- 262
- Avg merge
- 9d 13h
- Merged PRs (30d)
- 8
Description
Hello, I'm running into some memory problems while reconstructing some data using ossart_tv. Normal ossart works fine and I've narrowed down the problem to the tvdenoise function (from _tv_proximal) called from utilities/im_3d_denoise.py. My goal is to reconstruct data into a 2000^3 volume. Other algorithms like fdk, sirt, os_pcsd, and os_asd_pocs works fine.
I use TIGRE version 3.0.0 in Python with the following versions:
- Python/3.10.4
- GCC/11.3.0
- CUDA/12.1.1
- cuDNN/8.9.2.26
Hardware (HPC):
- CPU: Intel Xeon Platinum 8470
- GPU: NVIDIA H100 80GB HBM3
I've tried the code with 1, 2, and 3 GPUs.
A minimal example which replicates im_3d_denoise.py:
```
import numpy as np
from _tv_proximal import tvdenoise
from tigre.utilities.gpu import GpuIds
for i in range(1000, 2100, 100):
print(f'{i = }')
img = np.random.rand(i, i, i).astype(np.float32)
gpuids=GpuIds()
lmbda=50
iter=50
print(f'Image memory consumption: {img.nbytes / 1024 / 1024 / 1024:.4f} GB.')
img = tvdenoise(img, iter, lmbda, gpuids)
```
This code produces the following error for some different combinations of cpu memory and number of GPUs:
**CPU memory 512GB, 1 GPU: Error at i=2000 - Image memory = 30GB**
Common/CUDA/TIGRE_common.cpp (18): tvDenoise:tvdenoising:Memory TV dneoising requires 5 times the image memory. Your GPU(s) do not have the required memory.
This memory will be attempted to allocate on the CPU, Whic may fail or slow the computation by a very significant amount.
**CPU memory 512GB, 2 GPUs: Error at i=1600 - Image memory = 16GB**
Common/CUDA/TIGRE_common.cpp (7): TV minimization
Common/CUDA/TIGRE_common.cpp (14): CBCT:CUDA:TVdenoising invalid argument
**CPU memory 512GB, 3 GPUs: Error at i=1600 - Image memory = 16GB**
Common/CUDA/TIGRE_common.cpp (7): TV minimization
Common/CUDA/TIGRE_common.cpp (14): CBCT:CUDA:TVdenoising invalid argument
**CPU memory 128GB, 3 GPUs: Error at i=1600 - Image memory = 16GB**
Common/CUDA/TIGRE_common.cpp (7): TV minimization
Common/CUDA/TIGRE_common.cpp (14): CBCT:CUDA:TVdenoising invalid argument
From my understanding I would need around 150GB (2000^3) of memory on the GPUs, i.e. 2 or even 3 GPUs should be enough (80GB each).
- Do you know why this happens? Maybe it makes sense, but my profficiency in CUDA code is not the best.
- Do you have any suggestions as to what I can try to fix this?
Contributor guide
Assessment
This issue has not been assessed yet.