facebookresearch / facebookresearch/faiss

Faiss Search on GPU Index Causes PyTorch to Fail to Coalesce Sparse CUDA Matrices

Open
#2,136 2 comments 1 reaction 1 assignee Claimed by @wickedfoo View on GitHub
GPU Integration unconfirmed-bug
Dominant language
C++
Stars
40.9k
Forks
4.5k
PR merge metrics
No merged PRs in 30d

Description

# Summary

# Platform

OS: Ubuntu 20.04

Faiss version: release 1.7.1 (cff072b)

Installed from: anaconda

Faiss compilation options: N/A

Running on:
- [ ] CPU
- [x] GPU

Interface:
- [ ] C++
- [x] Python

# Reproduction instructions

The following script should reproduce the error:

```
import torch
import faiss
import numpy as np

indices = torch.tensor([[1, 0], [2, 3], [2, 4], [3, 2], [4, 1]], dtype=torch.long).T
values = torch.tensor([1., 2., 3., 4., 5.])

sparse = torch.sparse_coo_tensor(indices, values, device='cuda', size=(5, 5))

# the following line will run
# sparse.coalesce()

random_sequence = np.random.randn(100, 5).astype(np.float32)

res = faiss.StandardGpuResources()
index = faiss.index_cpu_to_gpu(res, 0, faiss.IndexFlatL2(5))
index.add(random_sequence)
# coalesce() will run below if no search is performed
D, I = index.search(random_sequence, 2)

# the following line won't run
res_2 = sparse.coalesce()
print(res_2)
```
PyTorch was able to coalesce() a sparse matrix on the GPU, but after a Faiss search on an unrelated array with an GPU Index, trying to run coalesce() on a CUDA sparse matrix produces the following error:
```
Traceback (most recent call last):
...
File "/mnt/[PATH]/pytorch_error.py", line 23, in
res_2 = sparse.coalesce()
RuntimeError: merge_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered
```
Environment:
pytorch 1.10.0
cudatoolkit 11.3.1
faiss-gpu 1.7.1
nvidia-driver-495

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.