cupy / cupy/cupy

cupy.linalg.svd on 'large matrices' leads to cusolver status invalid value error

Open
#7,065 5 comments 1 reaction 0 assignees View on GitHub
issue-checked
Dominant language
Python
Stars
12.3k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
45

Description

### Description

Hi all,

cupy linalg svd is giving me a _CUSOLVERError: CUSOLVER_STATUS_INVALID_VALUE_ error for 'large' matrices, e.g. of size 100,000 x 100. I didn't use to have these errors on the previous cupy version I was running. I updated the cupy package some weeks ago.

Weirdly enough, 'small' matrices, e.g. 10,000 x 100 give no problems. Running on an NVIDIA A6000, so memory should not be the problem. Couldn't find any recent bug reporting on cupy linalg svd. Someone an idea on further investigations to find out what the cause might be?

Thanks in advance,

Roger

### To Reproduce

```py
import cupy
a = cupy.random.rand(100_000,100)
b = cupy.linalg.svd(a, full_matrices=False, compute_uv=True)
```

### Installation

Conda-Forge (`conda install ...`)

### Environment

```
OS : Linux-5.15.0-48-generic-x86_64-with-glibc2.35
Python Version : 3.9.13
CuPy Version : 11.1.0
CuPy Platform : NVIDIA CUDA
NumPy Version : 1.23.1
SciPy Version : 1.8.0
Cython Build Version : 0.29.32
Cython Runtime Version : None
CUDA Root : /home/roger/miniconda3
nvcc PATH : /home/roger/miniconda3/bin/nvcc
CUDA Build Version : 10020
CUDA Driver Version : 11070
CUDA Runtime Version : 10020
cuBLAS Version : (available)
cuFFT Version : 10102
cuRAND Version : 10102
cuSOLVER Version : (10, 3, 0)
cuSPARSE Version : (available)
NVRTC Version : (10, 2)
Thrust Version : 100907
CUB Build Version :
Jitify Build Version : 3ecec55
cuDNN Build Version : None
cuDNN Version : None
NCCL Build Version : None
NCCL Runtime Version : None
cuTENSOR Version : None
cuSPARSELt Build Version : None
Device 0 Name : NVIDIA RTX A6000
Device 0 Compute Capability : 86
Device 0 PCI Bus ID : 0000:73:00.0

```

### Additional Information

The full error output is:

```
---------------------------------------------------------------------------
CUSOLVERError Traceback (most recent call last)
Input In [13], in ()
----> 1 b = cupy.linalg.svd(a, full_matrices=False, compute_uv=True)

File ~/miniconda3/lib/python3.9/site-packages/cupy/linalg/_decomposition.py:566, in svd(a, full_matrices, compute_uv)
564 rwork = cupy.empty(min(m, n)-1, dtype=s_dtype)
565 rwork_ptr = rwork.data.ptr
--> 566 gesvd(
567 handle, job_u, job_vt, m, n, x.data.ptr, m, s.data.ptr, u_ptr, m,
568 vt_ptr, n, workspace.data.ptr, buffersize, rwork_ptr,
569 dev_info.data.ptr)
570 cupy.linalg._util._check_cusolver_dev_info_if_synchronization_allowed(
571 gesvd, dev_info)
573 s = s.astype(s_dtype, copy=False)

File cupy_backends/cuda/libs/cusolver.pyx:2731, in cupy_backends.cuda.libs.cusolver.dgesvd()

File cupy_backends/cuda/libs/cusolver.pyx:2740, in cupy_backends.cuda.libs.cusolver.dgesvd()

File cupy_backends/cuda/libs/cusolver.pyx:1079, in cupy_backends.cuda.libs.cusolver.check_status()

CUSOLVERError: CUSOLVER_STATUS_INVALID_VALUE

```

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure with the supplied 100,000 x 100 example and compare it with the working 10,000 x 100 case. Inspect cupy.linalg.svd in cupy/linalg/_decomposition.py around line 566, then follow the dgesvd binding in cupy_backends/cuda/libs/cusolver.pyx around lines 2731-2740. Done means the cause is confirmed and the large-matrix case has a tested resolution or documented compatibility explanation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
hpc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.