AnswerDotAI / AnswerDotAI/RAGatouille

[HELP] Can Ragatouille take chunks with their embeddings as input for indexing?

Open
#264 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4k
Forks
276
PR merge metrics
No merged PRs in 30d

Description

- I want to index a long document with muti-vector embeddings.
- Say I have span annotation for how I want my document to be chunked and want to apply late chunking to it, that is i want to have all the embeddings of my tokens before I perform the chunking.
- This way I have contextualized chunks with multi-vector representation.
Essentially what I want to do is Late Chunking without the chunk-wise vector pooling at the end

How do I do it with ragatouille?
I know how to get multi-vector embeddings for my document but can't figure out how to index and leverage the optimizations provided by Ragatouille.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named. Start by reviewing RAGatouille's indexing entry points and documentation to determine whether pre-chunked multi-vector embeddings are accepted; done means establishing whether late chunking without final vector pooling can be indexed and used with Ragatouille's retrieval optimizations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.