arrayfire / arrayfire/arrayfire
[BUG] sparse matrix multiplication causes access violation
- Dominant language
- C++
- Stars
- 4.9k
- Forks
- 555
- Avg merge
- 1h 24m
- Merged PRs (30d)
- 1
Description
Creating a sparse matrix and then multiplying with same or other sparse matrix or with another matrix causes crash and access violation.
Description
===========
* Did you build ArrayFire yourself or did you use the official installers:
use official installer
* Which backend is experiencing this issue? (CPU, CUDA, OpenCL)
CUDA
* Do you have a workaround?
NO
* Can the bug be reproduced reliably on your system?
YES
* A clear and concise description of what you expected to happen.
* Run your executable with AF_TRACE=all and AF_PRINT_ERRORS=1 environment
variables set.
* Screenshot or terminal output of the results
see below
Reproducible Code and/or Steps
------------------------------
I am running the following code n GOOGLE tests, However, I am having trouble with sparse matrices in other places as well. Similar nature.
TEST(TestSuiteName, TestArrayFire)
{
float v[] = {5, 8, 3, 6};
int r[] = {0, 0, 2, 3, 4};
int c[] = {0, 1, 2, 1};
const int M = 4, N = 4, nnz = 4;
array vals = array(dim4(nnz), v);
array row_ptr = array(dim4(M + 1), r);
array col_idx = array(dim4(nnz), c);
af_print(vals);
af_print(row_ptr);
af_print(col_idx);
// Create sparse array (CSR) from af::arrays containing values,
// row pointers, and column indices.
array sparse = af::sparse(M, N, vals, row_ptr, col_idx, AF_STORAGE_CSR);
af_print(sparse);
auto res = sparse * sparse;
af_print(res);
}
StackTrace when debugging:
-------------------------------
af.dll!af_get_last_error(char * * str, __int64 * len) Line 46 (f:\buildbot\worker\win10-cuda-installer\build\src\api\unified\error.cpp:46)
af.dll!af::operator*(const af::array & lhs, const af::array & rhs) Line 939 (f:\buildbot\worker\win10-cuda-installer\build\src\api\cpp\array.cpp:939)
Output:
-------------------
Loaded 'C:\Windows\System32\dcomp.dll'.
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(361) ] Default device: 0
[unified][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\api\unified\symbol_manager.cpp(153) ] Device Count: 4.
[unified][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\api\unified\symbol_manager.cpp(143) ] Attempting: Default System Paths
Loaded 'C:\Programs\ArratFire\v3\lib\afcuda.dll'. Symbols loaded.
Loaded 'C:\Windows\System32\nvcuda.dll'.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cublas64_12.dll'. Module was built without symbols.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cufft64_11.dll'. Module was built without symbols.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cusolver64_11.dll'. Module was built without symbols.
2
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cusparse64_12.dll'. Module was built without symbols.
Unloaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cusparse64_12.dll'.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\nvrtc64_120_0.dll'. Module was built without symbols.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\cublasLt64_12.dll'. Module was built without symbols.
Loaded 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.2\bin\nvJitLink_120_0.dll'. Module was built without symbols.
[unified][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\api\unified\symbol_manager.cpp(146) ] Found: afcuda.dll
Loaded 'C:\Windows\System32\DriverStore\FileRepository\nvdmi.inf_amd64_893ed8ff453738db\nvcuda64.dll'.
[unified][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\api\unified\symbol_manager.cpp(153) ] Device Count: 1.
[unified][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\api\unified\symbol_manager.cpp(208) ] AF_DEFAULT_BACKEND: cuda
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\common\DependencyModule.cpp(101) ] Attempting to load: forge.dll
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\common\DependencyModule.cpp(104) ] Found: forge.dll
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(494) ] CUDA Driver supports up to CUDA 12.2.0 ArrayFire CUDA Runtime 12.0.0
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(479) ] CUDA driver version(12.2.0) not part of the CudaToDriverVersion array. Please create an issue or a pull request on the ArrayFire repository to update the CudaToDriverVersion variable with this version of the CUDA runtime.
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(562) ] Found 1 CUDA devices
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(590) ] Found device: NVIDIA GeForce GTX 1050 Ti with Max-Q Design (sm_61) (4 GB | ~2076.416015625 GFLOPs | 6 SMs)
[platform][1688279120][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(625) ] AF_CUDA_DEFAULT_DEVICE:
[platform][1688279121][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\device_manager.cpp(644) ] Default device: 0(NVIDIA GeForce GTX 1050 Ti with Max-Q Design)
[mem][1688279121][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\common\DefaultMemoryManager.cpp(128) ] memory[0].max_bytes: 3 GB
[mem][1688279121][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\memory.cpp(157) ] nativeAlloc: 1 KB 0x1001000000
[mem][1688279121][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\memory.cpp(157) ] nativeAlloc: 1 KB 0x1001000400
[mem][1688279121][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\memory.cpp(157) ] nativeAlloc: 1 KB 0x1001000800
als
[4 1 1 1]
[mem][1688279137][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\memory.cpp(157) ] nativeAlloc: 1 KB 0x1001000c00
[jit][1688279137][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\compile_module.cpp(466) ] {291210446400920389 : loaded from C:\Users\rohrb\AppData\Local\Temp\\ArrayFire\KER291210446400920389_CU_61_AF_38.bin for NVIDIA GeForce GTX 1050 Ti with Max-Q Design }
[kernel][1688279137][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
5.0000
8.0000
3.0000
6.0000
row_ptr
[5 1 1 1]
[jit][1688279139][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\compile_module.cpp(466) ] {9963207849209086996 : loaded from C:\Users\rohrb\AppData\Local\Temp\\ArrayFire\KER9963207849209086996_CU_61_AF_38.bin for NVIDIA GeForce GTX 1050 Ti with Max-Q Design }
[kernel][1688279139][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
2
0
2
3
4
col_idx
[4 1 1 1]
[kernel][1688279147][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
0
1
2
1
sparse
Storage Format : AF_STORAGE_CSR
[4 4 1 1]
sparse: Values
[4 1 1 1]
[kernel][1688279152][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
5.0000
8.0000
3.0000
6.0000
sparse: RowIdx
[5 1 1 1]
[kernel][1688279152][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
2
0
2
3
4
sparse: ColIdx
[4 1 1 1]
[kernel][1688279152][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\cuda\Kernel.hpp(42) ] Launching arrayfire::cuda::transpose: Blocks: [1, 1, 1] Threads: [32, 8, 1] Shared Memory: 0
0
1
2
1
Exception thrown at 0x00007FFF04ADADCC (af.dll) in runTests.exe: 0xC0000005: Access violation reading location 0xFFFFFFFFFFFFFFFF.
unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.
unknown file: error: SEH exception with code 0xc0000005 thrown in the test body.
System Information
------------------
Please provide the following information:
1. ArrayFire version
ArrayFire v3.8.3
2. Devices installed on the system
2 Graphics Cards 1 CPU
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\common\DependencyModule.cpp(104) ] Found: forge.dll
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(217) ] Found 5 OpenCL platforms
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(229) ] Found 1 devices on platform NVIDIA CUDA
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(234) ] Found device NVIDIA GeForce GTX 1050 Ti with Max-Q Design on platform NVIDIA CUDA
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(229) ] Found 1 devices on platform Intel(R) OpenCL HD Graphics
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(234) ] Found device Intel(R) UHD Graphics 630 on platform Intel(R) OpenCL HD Graphics
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(229) ] Found 1 devices on platform Intel(R) OpenCL
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(234) ] Found device Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz on platform Intel(R) OpenCL
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(229) ] Found 1 devices on platform Intel(R) FPGA Emulation Platform for OpenCL(TM)
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(234) ] Found device Intel(R) FPGA Emulation Device on platform Intel(R) FPGA Emulation Platform for OpenCL(TM)
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(229) ] Found 0 devices on platform Intel(R) FPGA SDK for OpenCL(TM)
[platform][1688279118][13192] [ F:\buildbot\worker\win10-cuda-installer\build\src\backend\opencl\device_manager.cpp(239) ] Found 4 OpenCL devices
4. (optional) Output from the af::info() function if applicable.
2023-07-01 22:53:14.745 info : Setting CUDA Backend
2023-07-01 22:53:14.745 info : Testing Backend:
2023-07-01 22:53:14.745 info : ArrayFire v3.8.3 (CUDA, 64-bit Windows, build 987d5675a)
Platform: CUDA Runtime 12.0, Driver: 12020
[0] NVIDIA GeForce GTX 1050 Ti with Max-Q Design, 4096 MB, CUDA Compute 6.1
5. Output from the following scripts:
Run one of the following commands based on your OS
Windows:
Download clinfo from https://github.com/Oblomov/clinfo
If you have NVIDIA GPUs. Run nvidia-smi usually located in
C:\Program Files\NVIDIA Corporation\NVSMI
Provide driver version for your GPU. (This is vendor specific)
536.40
Checklist
---------
- [x] Using the latest available ArrayFire release
- [x] GPU drivers are up to date
Contributor guide
Assessment
This issue has not been assessed yet.