google-research / google-research/google-research

[SCANN] support for large (out-of-memory) dataset

Open
#1,237 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
38.8k
Forks
8.5k
PR merge metrics
No merged PRs in 30d

Description

I wonder if SCANN supports large dataset in general. For example, to train the sift1B (1 billion x 128 dim) dataset on a small machine with 64 GB of memory, I need to memmap the dataset and pass it to the scann's builder. However, seems SCANN only supports to load the dataset fully in memory before it can start training.

Is there any way to train large dataset given constraint memory (like faiss)? Or is SCANN designed for rather small datasets only?

```
Traceback (most recent call last):
File "/data/scann_test/scann_sift1B.py", line 68, in
searcher = scann.scann_ops_pybind.builder(xb, topK, "squared_l2").tree(
File "/home/ubuntu/anaconda3/envs/scann/lib/python3.9/site-packages/scann/scann_ops/py/scann_builder.py", line 243, in build
return self.builder_lambda(self.db, config, self.training_threads, **kwargs)
File "/home/ubuntu/anaconda3/envs/scann/lib/python3.9/site-packages/scann/scann_ops/py/scann_ops_pybind.py", line 78, in builder_lambda
return create_searcher(db, config, training_threads, **kwargs)
File "/home/ubuntu/anaconda3/envs/scann/lib/python3.9/site-packages/scann/scann_ops/py/scann_ops_pybind.py", line 86, in create_searcher
scann_pybind.ScannNumpy(db, scann_config, training_threads))
TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
1. scann_pybind.ScannNumpy(arg0: str, arg1: str)
2. scann_pybind.ScannNumpy(arg0: numpy.ndarray[numpy.float32], arg1: str, arg2: int)

Invoked with: memmap([[2.9440201e-20, 1.8216880e-44, 2.8302559e+26, ..., 3.4590261e-11,
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.