google-research / google-research/google-research
[SCANN] support for large (out-of-memory) dataset
- Dominant language
- Jupyter Notebook
- Stars
- 38.8k
- Forks
- 8.5k
- PR merge metrics
- No merged PRs in 30d
Description
I wonder if SCANN supports large dataset in general. For example, to train the sift1B (1 billion x 128 dim) dataset on a small machine with 64 GB of memory, I need to memmap the dataset and pass it to the scann's builder. However, seems SCANN only supports to load the dataset fully in memory before it can start training.
Is there any way to train large dataset given constraint memory (like faiss)? Or is SCANN designed for rather small datasets only?
```
Traceback (most recent call last):
File "/data/scann_test/scann_sift1B.py", line 68, in
searcher = scann.scann_ops_pybind.builder(xb, topK, "squared_l2").tree(
File "/home/ubuntu/anaconda3/envs/scann/lib/python3.9/site-packages/scann/scann_ops/py/scann_builder.py", line 243, in build
return self.builder_lambda(self.db, config, self.training_threads, **kwargs)
File "/home/ubuntu/anaconda3/envs/scann/lib/python3.9/site-packages/scann/scann_ops/py/scann_ops_pybind.py", line 78, in builder_lambda
return create_searcher(db, config, training_threads, **kwargs)
File "/home/ubuntu/anaconda3/envs/scann/lib/python3.9/site-packages/scann/scann_ops/py/scann_ops_pybind.py", line 86, in create_searcher
scann_pybind.ScannNumpy(db, scann_config, training_threads))
TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
1. scann_pybind.ScannNumpy(arg0: str, arg1: str)
2. scann_pybind.ScannNumpy(arg0: numpy.ndarray[numpy.float32], arg1: str, arg2: int)
Invoked with: memmap([[2.9440201e-20, 1.8216880e-44, 2.8302559e+26, ..., 3.4590261e-11,
```
Contributor guide
Assessment
This issue has not been assessed yet.