AnswerDotAI / AnswerDotAI/RAGatouille

Support exporting index to HuggingFace Hub

Open
#37 1 comment 1 reaction 0 assignees View on GitHub
enhancement good first issue help wanted
Dominant language
Python
Stars
4k
Forks
276
PR merge metrics
No merged PRs in 30d

Description

Indexing is time consuming, and oftentimes people would like to be able to easily share pre-built index for various common datasets, for general domain application (wikipedia, code documentation...) and evaluation purposes.

A simple way to support this would be to add a `util` function that'd export the full index folder to the huggingface model, effectively exporting both the ColBERT config + the compressed vectors, allowing you to to do something like `RAGPretrainedModel.from_prebuilt_index("EXAMPLE_USER/Wikipedia")` and immediately begin querying the index.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the utility functions that create or load a full index folder and the RAGPretrainedModel.from_prebuilt_index entry point. Check how ColBERT configuration and compressed vectors are stored; done means an index can be exported to HuggingFace Hub and loaded from a repository such as EXAMPLE_USER/Wikipedia for querying.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.