AnswerDotAI / AnswerDotAI/RAGatouille

Training without labels - adapting existing pretrained backbone

Open
#240 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4k
Forks
276
PR merge metrics
No merged PRs in 30d

Description

Use case:
- A pretrained model is available for my domain (encoder transformer, but no existing colbert model, vastly different tokenizer etc)
- I would like to initialize a new colbert/ ragatouille model with it as described in #206

Question:
1. Can I provide training data not labelled or in pairs/ triplets? What would be the recommended way to freeze the pretrained backbone and only train the linear projection + k-means sections?
2. Is there a way to decouple the initial trained backbone embedding from this? I.e. can I embed my unsupervised training corpus up front with my backbone model and then use those embeddings as my training set, instead of raw "text" input?
a. use case is to test many configurations of the downstream projection layer + k means indexing, with the aim of reducing the encoding GPU costs

PS. big fan of this + colbert work in general!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading the approach described in issue #206 and review how RAGatouille handles pretrained backbones, training data, linear projection, and k-means sections. Determine whether unlabeled or precomputed embeddings can be supported and what changes would be needed; the issue provides no named files or tests to run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.