facebookresearch / facebookresearch/faiss
Faiss Search on GPU Index Causes PyTorch to Fail to Coalesce Sparse CUDA Matrices
- Dominant language
- C++
- Stars
- 40.9k
- Forks
- 4.5k
- PR merge metrics
- No merged PRs in 30d
Description
# Summary
# Platform
OS: Ubuntu 20.04
Faiss version: release 1.7.1 (cff072b)
Installed from: anaconda
Faiss compilation options: N/A
Running on:
- [ ] CPU
- [x] GPU
Interface:
- [ ] C++
- [x] Python
# Reproduction instructions
The following script should reproduce the error:
```
import torch
import faiss
import numpy as np
indices = torch.tensor([[1, 0], [2, 3], [2, 4], [3, 2], [4, 1]], dtype=torch.long).T
values = torch.tensor([1., 2., 3., 4., 5.])
sparse = torch.sparse_coo_tensor(indices, values, device='cuda', size=(5, 5))
# the following line will run
# sparse.coalesce()
random_sequence = np.random.randn(100, 5).astype(np.float32)
res = faiss.StandardGpuResources()
index = faiss.index_cpu_to_gpu(res, 0, faiss.IndexFlatL2(5))
index.add(random_sequence)
# coalesce() will run below if no search is performed
D, I = index.search(random_sequence, 2)
# the following line won't run
res_2 = sparse.coalesce()
print(res_2)
```
PyTorch was able to coalesce() a sparse matrix on the GPU, but after a Faiss search on an unrelated array with an GPU Index, trying to run coalesce() on a CUDA sparse matrix produces the following error:
```
Traceback (most recent call last):
...
File "/mnt/[PATH]/pytorch_error.py", line 23, in
res_2 = sparse.coalesce()
RuntimeError: merge_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered
```
Environment:
pytorch 1.10.0
cudatoolkit 11.3.1
faiss-gpu 1.7.1
nvidia-driver-495
Contributor guide
Assessment
This issue has not been assessed yet.