huggingface / huggingface/lighteval
[FT] Evaluation using a multi-document RAG based on statistical tools and LLM as judge
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Issue encountered
It would be good to have a system for evaluating both the relevance of the RAG and its use by the LLM in producing the response. My first intuition would be a multi-stage system with, on the one hand, an evaluation in the manner of Sentence Transformers' information-retrieval evaluator (@tomaarsen), and on the other hand, an evaluation with certain metrics of the model's ability to interpret the documents from the context provided by the RAG system (mutli-documents) in order to test :
- Answer Relevancy
- Faithfulness
- Contextual Precision
- Contextual Recall
- Contextual Relevancy
- Tool Correctness
- Hallucination
## Solution/Feature
For the moment, it seems to me that the simplest way to implement this system is to combine two tools, sentence-transformers and DeepEval, and then aggregate the results in the form of a synthetic indicator, but this method requires a multitude of libraries and a connection between the database necessary for the sentence-transformers and DeepEval evaluators.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.