deepset-ai / deepset-ai/haystack-core-integrations

Allowing other LLMs and custom prompts in evaluation (specifically, deepeval)

Open
#1,872 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

integration:deepeval P3
Dominant language
Python
Stars
203
Forks
332
Avg merge
2d 4h
Merged PRs (30d)
80

Description

Is your feature request related to a problem? Please describe.
I cannot use a (small) local LLM or customized prompts for evaluation of the RAG pipeline output. Smaller LLMs (e.g., minicheck) have become as good as GPT for evaluation.

Describe the solution you'd like
I would like to use a small local LLM for evaluation of the RAG pipeline output. At this time, it seems that only GPT LLMs are allowed. Smaller LLMs (e.g., minicheck) have become as good as GPT for evaluation. These local LLMs are available via Ollama. Also, there does not seem to be a way to customize the prompts used in haystack-deepeval.

Describe alternatives you've considered
Use deepeval "offline", i.e. saved the question, contexts (chunks) and answer and use deepeval locally. This is not very convenient, since I would like to be able to fine tune the model.

Additional context
The ability to use deepeval to evaluate a model during fine tuning is very useful. It is also good to be able to customize the prompt, since it looks like CoT or other techniques can improve evaluation outputs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the haystack-deepeval integration and reviewing how it currently selects GPT models and builds evaluation prompts. Compare that flow with deepeval's local Ollama usage; done means a local model and customized prompts can be used during RAG evaluation, with the supported configuration documented or covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, python
Domain
ai, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.