AnswerDotAI / AnswerDotAI/RAGatouille
Training without labels - adapting existing pretrained backbone
- Dominant language
- Python
- Stars
- 4k
- Forks
- 276
- PR merge metrics
- No merged PRs in 30d
Description
Use case:
- A pretrained model is available for my domain (encoder transformer, but no existing colbert model, vastly different tokenizer etc)
- I would like to initialize a new colbert/ ragatouille model with it as described in #206
Question:
1. Can I provide training data not labelled or in pairs/ triplets? What would be the recommended way to freeze the pretrained backbone and only train the linear projection + k-means sections?
2. Is there a way to decouple the initial trained backbone embedding from this? I.e. can I embed my unsupervised training corpus up front with my backbone model and then use those embeddings as my training set, instead of raw "text" input?
a. use case is to test many configurations of the downstream projection layer + k means indexing, with the aim of reducing the encoding GPU costs
PS. big fan of this + colbert work in general!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the approach described in issue #206 and review how RAGatouille handles pretrained backbones, training data, linear projection, and k-means sections. Determine whether unlabeled or precomputed embeddings can be supported and what changes would be needed; the issue provides no named files or tests to run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100