AnswerDotAI / AnswerDotAI/RAGatouille

Save the ColBERT encodings to disk.

Open
#237 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4k
Forks
276
PR merge metrics
No merged PRs in 30d

Description

I have a use case where I run ColBERT on CPU on a couple thousand documents. For this I don't use PLAID but the `encode` and `search_encoded_docs` methods and the search works fast enough, the problem is that encoding all these documents on CPU takes time and I don't want to encode everything everytime I deploy the model so I developed a way for saving and loading these encodings:

https://github.com/ChatFAQ/ChatFAQ/blob/cc19e4b85198062888d6320e59276db31461f4e9/chat_rag/chat_rag/retrievers/colbert_retriever.py#L163

If interested I could improve and integrate this into the `RAGPretrainedModel` or `ColBERT` classes and make a PR.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the referenced chat_rag/retrievers/colbert_retriever.py implementation around line 163, then inspect the encode and search_encoded_docs methods in RAGPretrainedModel or ColBERT. The issue is complete when encodings can be saved and loaded for later searches without re-encoding the documents.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, search
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.