RuntimeError: Error building extension 'decompress_residuals_cpp' - ninja/Colbert/Torch error.

Got an unexpected error while running an example in a VSCode Jupyter Notebook (Python3.10, headless remote Ubuntu 22.04 server). Tried recreating the environment, without results.

Code:
```python
import ragatouille as rag
RAG = rag.RAGPretrainedModel.from_pretrained("colbert-ir/colbertv2.0")
arrakis_data = get_wikipedia_page("Arrakis")
RAG.index( #this line fails
collection=[arrakis_data],
document_ids=['arrakis'],
document_metadatas=[{"entity": "planet", "source": "wikipedia"}],
index_name="Arrakis",
max_document_length=180,
split_documents=True
)
```

Output:

```
________________________________________________________________________________
WARNING! You have a GPU available, but only `faiss-cpu` is currently installed.
This means that indexing will be slow. To make use of your GPU.
Please install `faiss-gpu` by running:
pip uninstall --y faiss-cpu & pip install faiss-gpu
________________________________________________________________________________
Will continue with CPU indexing in 5 seconds...

[Mar 05, 09:11:10] #> Note: Output directory .ragatouille/colbert/indexes/Arrakis already exists

[Mar 05, 09:11:10] #> Will delete 1 files already at .ragatouille/colbert/indexes/Arrakis in 20 seconds...
[Mar 05, 09:11:32] [0] #> Encoding 27 passages..
[Mar 05, 09:11:32] [0] avg_doclen_est = 133.8148193359375 len(local_sample) = 27
[Mar 05, 09:11:32] [0] Creating 512 partitions.
[Mar 05, 09:11:32] [0] *Estimated* 3,613 embeddings.
[Mar 05, 09:11:32] [0] #> Saving the indexing plan to .ragatouille/colbert/indexes/Arrakis/plan.json ..
Clustering 3433 points in 128D to 512 clusters, redo 1 times, 20 iterations
Preprocessing in 0.00 s
Iteration 14 (0.19 s, search 0.18 s): objective=749.307 imbalance=1.523 nsplit=0
WARNING clustering 3433 points to 512 centroids: please provide at least 19968 training points
Iteration 19 (0.25 s, search 0.24 s): objective=749.307 imbalance=1.523 nsplit=0
[Mar 05, 09:11:33] Loading decompress_residuals_cpp extension (set COLBERT_LOAD_TORCH_EXTENSION_VERBOSE=True for more info)...
```
then:
```
---------------------------------------------------------------------------
CalledProcessError Traceback (most recent call last)
File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2096](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2096), in _run_ninja_build(build_directory, verbose, error_prefix)
[2095](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2095) stdout_fileno = 1
-> [2096](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2096) subprocess.run(
[2097](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2097) command,
[2098](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2098) stdout=stdout_fileno if verbose else subprocess.PIPE,
[2099](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2099) stderr=subprocess.STDOUT,
[2100](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2100) cwd=build_directory,
[2101](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2101) check=True,
[2102](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2102) env=env)
[2103](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2103) except subprocess.CalledProcessError as e:
[2104](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2104) # Python 2 and 3 compatible way of getting the error object.

File /usr/lib/python3.10/subprocess.py:526, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
[525](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/usr/lib/python3.10/subprocess.py:525) if check and retcode:
--> [526](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/usr/lib/python3.10/subprocess.py:526) raise CalledProcessError(retcode, process.args,
[527](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/usr/lib/python3.10/subprocess.py:527) output=stdout, stderr=stderr)
[528](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/usr/lib/python3.10/subprocess.py:528) return CompletedProcess(process.args, retcode, stdout, stderr)

CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

RuntimeError Traceback (most recent call last)
Cell In[4], [line 1](vscode-notebook-cell:?execution_count=4&line=1)
----> [1](vscode-notebook-cell:?execution_count=4&line=1) RAG.index(
[2](vscode-notebook-cell:?execution_count=4&line=2) collection=[arrakis_data],
[3](vscode-notebook-cell:?execution_count=4&line=3) document_ids=['arrakis'],
[4](vscode-notebook-cell:?execution_count=4&line=4) document_metadatas=[{"entity": "planet", "source": "wikipedia"}],
[5](vscode-notebook-cell:?execution_count=4&line=5) index_name="Arrakis",
[6](vscode-notebook-cell:?execution_count=4&line=6) max_document_length=180,
[7](vscode-notebook-cell:?execution_count=4&line=7) split_documents=True
[8](vscode-notebook-cell:?execution_count=4&line=8) )

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:210](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:210), in RAGPretrainedModel.index(self, collection, document_ids, document_metadatas, index_name, overwrite_index, max_document_length, split_documents, document_splitter_fn, preprocessing_fn, bsize)
[201](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:201) document_splitter_fn = None
[202](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:202) collection, pid_docid_map, docid_metadata_map = self._process_corpus(
[203](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:203) collection,
[204](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:204) document_ids,
(...)
[208](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:208) max_document_length,
[209](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:209) )
--> [210](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:210) return self.model.index(
[211](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:211) collection,
[212](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:212) pid_docid_map=pid_docid_map,
[213](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:213) docid_metadata_map=docid_metadata_map,
[214](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:214) index_name=index_name,
[215](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:215) max_document_length=max_document_length,
[216](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:216) overwrite=overwrite_index,
[217](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:217) bsize=bsize,
[218](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/RAGPretrainedModel.py:218) )

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:368](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:368), in ColBERT.index(self, collection, pid_docid_map, docid_metadata_map, index_name, max_document_length, overwrite, bsize)
[362](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:362) self.indexer = Indexer(
[363](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:363) checkpoint=self.checkpoint,
[364](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:364) config=self.config,
[365](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:365) verbose=self.verbose,
[366](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:366) )
[367](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:367) self.indexer.configure(avoid_fork_if_possible=True)
--> [368](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:368) self.indexer.index(
[369](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:369) name=self.index_name, collection=self.collection, overwrite=overwrite
[370](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:370) )
[372](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:372) self.index_path = str(
[373](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:373) Path(self.run_config.root)
[374](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:374) / Path(self.run_config.experiment)
[375](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:375) / "indexes"
[376](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:376) / self.index_name
[377](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:377) )
[378](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:378) self.config.root = str(
[379](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:379) Path(self.run_config.root) / Path(self.run_config.experiment) / "indexes"
[380](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/ragatouille/models/colbert.py:380) )

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:80](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:80), in Indexer.index(self, name, collection, overwrite)
[77](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:77) self.erase()
[79](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:79) if index_does_not_exist or overwrite != 'reuse':
---> [80](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:80) self.__launch(collection)
[82](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:82) return self.index_path

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:89](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:89), in Indexer.__launch(self, collection)
[87](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:87) shared_queues = []
[88](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:88) shared_lists = []
---> [89](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:89) launcher.launch_without_fork(self.config, collection, shared_lists, shared_queues, self.verbose)
[91](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:91) return
[93](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexer.py:93) manager = mp.Manager()

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:93](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:93), in Launcher.launch_without_fork(self, custom_config, *args)
[90](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:90) assert (custom_config.avoid_fork_if_possible or self.run_config.avoid_fork_if_possible)
[92](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:92) new_config = type(custom_config).from_existing(custom_config, self.run_config, RunConfig(rank=0))
---> [93](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:93) return_val = run_process_without_mp(self.callee, new_config, *args)
[95](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:95) return return_val

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:109](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:109), in run_process_without_mp(callee, config, *args)
[106](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:106) os.environ["CUDA_VISIBLE_DEVICES"] = ','.join(map(str, config.gpus_[:config.nranks]))
[108](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:108) with Run().context(config, inherit_config=False):
--> [109](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:109) return_val = callee(config, *args)
[110](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:110) torch.cuda.empty_cache()
[111](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/infra/launcher.py:111) return return_val

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:33](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:33), in encode(config, collection, shared_lists, shared_queues, verbose)
[31](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:31) def encode(config, collection, shared_lists, shared_queues, verbose: int = 3):
[32](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:32) encoder = CollectionIndexer(config=config, collection=collection, verbose=verbose)
---> [33](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:33) encoder.run(shared_lists)

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:68](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:68), in CollectionIndexer.run(self, shared_lists)
[65](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:65) print_memory_stats(f'RANK:{self.rank}')
[67](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:67) if not self.config.resume or not self.saver.try_load_codec():
---> [68](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:68) self.train(shared_lists) # Trains centroids from selected passages
[69](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:69) distributed.barrier(self.rank)
[70](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:70) print_memory_stats(f'RANK:{self.rank}')

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:237](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:237), in CollectionIndexer.train(self, shared_lists)
[234](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:234) print_memory_stats(f'RANK:{self.rank}')
[235](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:235) del sample
--> [237](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:237) bucket_cutoffs, bucket_weights, avg_residual = self._compute_avg_residual(centroids, heldout)
[239](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:239) if self.verbose > 1:
[240](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:240) print_message(f'avg_residual = {avg_residual}')

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:315](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:315), in CollectionIndexer._compute_avg_residual(self, centroids, heldout)
[314](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:314) def _compute_avg_residual(self, centroids, heldout):
--> [315](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:315) compressor = ResidualCodec(config=self.config, centroids=centroids, avg_residual=None)
[317](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:317) heldout_reconstruct = compressor.compress_into_codes(heldout, out_device='cuda' if self.use_gpu else 'cpu')
[318](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/collection_indexer.py:318) heldout_reconstruct = compressor.lookup_centroids(heldout_reconstruct, out_device='cuda' if self.use_gpu else 'cpu')

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:24](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:24), in ResidualCodec.__init__(self, config, centroids, avg_residual, bucket_cutoffs, bucket_weights)
[21](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:21) def __init__(self, config, centroids, avg_residual=None, bucket_cutoffs=None, bucket_weights=None):
[22](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:22) self.use_gpu = config.total_visible_gpus > 0
---> [24](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:24) ResidualCodec.try_load_torch_extensions(self.use_gpu)
[26](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:26) if self.use_gpu > 0:
[27](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:27) self.centroids = centroids.cuda().half()

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:103](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:103), in ResidualCodec.try_load_torch_extensions(cls, use_gpu)
[100](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:100) return
[102](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:102) print_message(f"Loading decompress_residuals_cpp extension (set COLBERT_LOAD_TORCH_EXTENSION_VERBOSE=True for more info)...")
--> [103](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:103) decompress_residuals_cpp = load(
[104](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:104) name="decompress_residuals_cpp",
[105](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:105) sources=[
[106](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:106) os.path.join(
[107](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:107) pathlib.Path(__file__).parent.resolve(), "decompress_residuals.cpp"
[108](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:108) ),
[109](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:109) os.path.join(
[110](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:110) pathlib.Path(__file__).parent.resolve(), "decompress_residuals.cu"
[111](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:111) ),
[112](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:112) ],
[113](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:113) verbose=os.getenv("COLBERT_LOAD_TORCH_EXTENSION_VERBOSE", "False") == "True",
[114](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:114) )
[115](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:115) cls.decompress_residuals = decompress_residuals_cpp.decompress_residuals_cpp
[117](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/residual.py:117) print_message(f"Loading packbits_cpp extension (set COLBERT_LOAD_TORCH_EXTENSION_VERBOSE=True for more info)...")

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1306](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1306), in load(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
[1214](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1214) def load(name,
[1215](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1215) sources: Union[str, List[str]],
[1216](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1216) extra_cflags=None,
(...)
[1224](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1224) is_standalone=False,
[1225](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1225) keep_intermediates=True):
[1226](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1226) """
[1227](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1227) Load a PyTorch C++ extension just-in-time (JIT).
[1228](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1228)
(...)
[1304](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1304) ... verbose=True)
[1305](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1305) """
-> [1306](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1306) return _jit_compile(
[1307](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1307) name,
[1308](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1308) [sources] if isinstance(sources, str) else sources,
[1309](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1309) extra_cflags,
[1310](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1310) extra_cuda_cflags,
[1311](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1311) extra_ldflags,
[1312](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1312) extra_include_paths,
[1313](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1313) build_directory or _get_build_directory(name, verbose),
[1314](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1314) verbose,
[1315](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1315) with_cuda,
[1316](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1316) is_python_module,
[1317](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1317) is_standalone,
[1318](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1318) keep_intermediates=keep_intermediates)

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1710](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1710), in _jit_compile(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
[1706](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1706) hipified_sources.add(hipify_result[s_abs].hipified_path if s_abs in hipify_result else s_abs)
[1708](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1708) sources = list(hipified_sources)
-> [1710](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1710) _write_ninja_file_and_build_library(
[1711](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1711) name=name,
[1712](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1712) sources=sources,
[1713](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1713) extra_cflags=extra_cflags or [],
[1714](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1714) extra_cuda_cflags=extra_cuda_cflags or [],
[1715](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1715) extra_ldflags=extra_ldflags or [],
[1716](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1716) extra_include_paths=extra_include_paths or [],
[1717](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1717) build_directory=build_directory,
[1718](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1718) verbose=verbose,
[1719](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1719) with_cuda=with_cuda,
[1720](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1720) is_standalone=is_standalone)
[1721](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1721) finally:
[1722](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1722) baton.release()

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1823](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1823), in _write_ninja_file_and_build_library(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_standalone)
[1821](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1821) if verbose:
[1822](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1822) print(f'Building extension module {name}...', file=sys.stderr)
-> [1823](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1823) _run_ninja_build(
[1824](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1824) build_directory,
[1825](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1825) verbose,
[1826](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:1826) error_prefix=f"Error building extension '{name}'")

File [~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2112](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2112), in _run_ninja_build(build_directory, verbose, error_prefix)
[2110](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2110) if hasattr(error, 'output') and error.output: # type: ignore[union-attr]
[2111](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2111) message += f": {error.output.decode(*SUBPROCESS_DECODE_ARGS)}" # type: ignore[union-attr]
-> [2112](https://vscode-remote+ssh-002dremote-002b192-002e168-002e100-002e166.vscode-resource.vscode-cdn.net/home/andrea.lorenzon/projects/rag-colbert/~/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/utils/cpp_extension.py:2112) raise RuntimeError(message) from e

RuntimeError: Error building extension 'decompress_residuals_cpp': [1/2] /usr/bin/nvcc --generate-dependencies-with-compile --dependency-output decompress_residuals.cuda.o.d -DTORCH_EXTENSION_NAME=decompress_residuals_cpp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/TH -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/THC -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/decompress_residuals.cu -o decompress_residuals.cuda.o
FAILED: decompress_residuals.cuda.o
/usr/bin/nvcc --generate-dependencies-with-compile --dependency-output decompress_residuals.cuda.o.d -DTORCH_EXTENSION_NAME=decompress_residuals_cpp -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1011\" -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/TH -isystem /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/torch/include/THC -isystem /usr/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' -std=c++17 -c /home/andrea.lorenzon/projects/rag-colbert/.venv/lib/python3.10/site-packages/colbert/indexing/codecs/decompress_residuals.cu -o decompress_residuals.cuda.o
/usr/include/c++/11/bits/std_function.h:435:145: error: parameter packs not expanded with ‘...’:
435 | function(_Functor&& __f)
| ^
/usr/include/c++/11/bits/std_function.h:435:145: note: ‘_ArgTypes’
/usr/include/c++/11/bits/std_function.h:530:146: error: parameter packs not expanded with ‘...’:
530 | operator=(_Functor&& __f)
| ^
/usr/include/c++/11/bits/std_function.h:530:146: note: ‘_ArgTypes’
ninja: build stopped: subcommand failed.
```
requirements.txt:
```
aiohttp==3.9.1
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.3.0
asttokens==2.4.1
async-timeout==4.0.3
attrs==23.2.0
bitarray==2.9.2
blinker==1.7.0
catalogue==2.0.10
certifi==2024.2.2
charset-normalizer==3.3.2
click==8.1.7
colbert-ai==0.2.19
comm==0.2.1
dataclasses-json==0.6.4
datasets==2.17.1
debugpy==1.8.1
decorator==5.1.1
Deprecated==1.2.14
dill==0.3.8
dirtyjson==1.0.8
distro==1.9.0
exceptiongroup==1.2.0
executing==2.0.1
faiss-cpu==1.8.0
faiss-gpu==1.7.2
filelock==3.13.1
Flask==3.0.2
frozenlist==1.4.1
fsspec==2023.10.0
git-python==1.0.3
gitdb==4.0.11
GitPython==3.1.42
greenlet==3.0.3
h11==0.14.0
httpcore==1.0.4
httpx==0.27.0
huggingface-hub==0.21.3
idna==3.6
ipykernel==6.29.3
ipython==8.22.1
itsdangerous==2.1.2
jedi==0.19.1
Jinja2==3.1.3
joblib==1.3.2
jsonpatch==1.33
jsonpointer==2.4
jupyter_client==8.6.0
jupyter_core==5.7.1
langchain==0.1.9
langchain-community==0.0.24
langchain-core==0.1.28
langsmith==0.1.10
llama-index==0.9.48
MarkupSafe==2.1.5
marshmallow==3.21.0
matplotlib-inline==0.1.6
mpmath==1.3.0
multidict==6.0.5
multiprocess==0.70.16
mypy-extensions==1.0.0
nest-asyncio==1.6.0
networkx==3.2.1
ninja==1.11.1.1
nltk==3.8.1
numpy==1.26.4
nvidia-cublas-cu12==12.1.3.1
nvidia-cuda-cupti-cu12==12.1.105
nvidia-cuda-nvrtc-cu12==12.1.105
nvidia-cuda-runtime-cu12==12.1.105
nvidia-cudnn-cu12==8.9.2.26
nvidia-cufft-cu12==11.0.2.54
nvidia-curand-cu12==10.3.2.106
nvidia-cusolver-cu12==11.4.5.107
nvidia-cusparse-cu12==12.1.0.106
nvidia-nccl-cu12==2.19.3
nvidia-nvjitlink-cu12==12.3.101
nvidia-nvtx-cu12==12.1.105
onnx==1.15.0
openai==1.13.3
orjson==3.9.15
packaging==23.2
pandas==2.2.1
parso==0.8.3
pexpect==4.9.0
pillow==10.2.0
platformdirs==4.2.0
prompt-toolkit==3.0.43
protobuf==4.25.3
psutil==5.9.8
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==15.0.0
pyarrow-hotfix==0.6
pydantic==2.6.3
pydantic_core==2.16.3
Pygments==2.17.2
python-dateutil==2.9.0
python-dotenv==1.0.1
pytz==2024.1
PyYAML==6.0.1
pyzmq==25.1.2
RAGatouille==0.0.7.post9
regex==2023.12.25
requests==2.31.0
ruff==0.1.15
safetensors==0.4.2
scikit-learn==1.4.1.post1
scipy==1.12.0
sentence-transformers==2.5.1
six==1.16.0
smmap==5.0.1
sniffio==1.3.1
SQLAlchemy==2.0.27
srsly==2.4.8
stack-data==0.6.3
sympy==1.12
tenacity==8.2.3
threadpoolctl==3.3.0
tiktoken==0.6.0
tokenizers==0.15.2
torch==2.2.1
tornado==6.4
tqdm==4.66.2
traitlets==5.14.1
transformers==4.38.2
triton==2.2.0
typing-inspect==0.9.0
typing_extensions==4.10.0
tzdata==2024.1
ujson==5.9.0
urllib3==2.2.1
voyager==2.0.2
wcwidth==0.2.13
Werkzeug==3.0.1
wrapt==1.16.0
xxhash==3.4.1
yarl==1.9.4

```

nvidia-smi:
```
Tue Mar 5 09:21:01 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.54.14 Driver Version: 550.54.14 CUDA Version: 12.4 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A4000 Off | 00000000:01:00.0 Off | Off |
| 41% 33C P8 15W / 140W | 1152MiB / 16376MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 1591 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 292316 C ...ojects/rag-colbert/.venv/bin/python 1138MiB |
+-----------------------------------------------------------------------------------------+
```

nvcc:
```
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Thu_Nov_18_09:45:30_PST_2021
Cuda compilation tools, release 11.5, V11.5.119
Build cuda_11.5.r11.5/compiler.30672275_0
```

Side note: if I install `faiss-gpu` instead of `faiss-cpu` as suggested, I get an `ImportError` from an `import faiss`.

Any suggestion would be great.

Start with the RAGPretrainedModel.index entry point shown at site-packages/ragatouille/RAGPretrainedModel.py:210 and trace the indexing call until decompress_residuals_cpp is loaded. Reproduce the VSCode Jupyter Notebook example on Ubuntu 22.04, capture the full ninja output, and establish a verified resolution for the extension build failure.

Written by the indexing model from the issue text.