AnswerDotAI / AnswerDotAI/RAGatouille

Question: How to get score of ranked document?

Open
#201 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
4k
Forks
276
PR merge metrics
No merged PRs in 30d

Description

Hey all, I'm using RAGatouille as a retriever for langchain like so

````Python
RAG = RAGPretrainedModel.from_index(".ragatouille/colbert/indexes/clr-docs")
retriever = RAG.as_langchain_retriever(k=10)
retrieval = RunnableParallel({
"context": retriever,
"question": RunnablePassthrough(),
})
rag_chain = (
retrieval
| RunnableParallel({
"answer": ANSWER_PROMPT | llm,
"docs": itemgetter("context")
})
)
````

I've traced down `as_langchain_retriever` but I could not find any helpful arguments.

How do I get the `score` for each retrieved doc so I can, i.e., calculate the auPRC score?

Thanks,
Bernhard.

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue points to RAG.as_langchain_retriever(k=10) and a LangChain retrieval chain, but names no repository file or test. Start by tracing that entry point and inspect the returned document interface and existing retrieval examples. Done means establishing a documented way to obtain each retrieved document's score for auPRC calculation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.