deepset-ai / deepset-ai/haystack-core-integrations
Allowing other LLMs and custom prompts in evaluation (specifically, deepeval)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 203
- Forks
- 332
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 80
Description
Is your feature request related to a problem? Please describe.
I cannot use a (small) local LLM or customized prompts for evaluation of the RAG pipeline output. Smaller LLMs (e.g., minicheck) have become as good as GPT for evaluation.
Describe the solution you'd like
I would like to use a small local LLM for evaluation of the RAG pipeline output. At this time, it seems that only GPT LLMs are allowed. Smaller LLMs (e.g., minicheck) have become as good as GPT for evaluation. These local LLMs are available via Ollama. Also, there does not seem to be a way to customize the prompts used in haystack-deepeval.
Describe alternatives you've considered
Use deepeval "offline", i.e. saved the question, contexts (chunks) and answer and use deepeval locally. This is not very convenient, since I would like to be able to fine tune the model.
Additional context
The ability to use deepeval to evaluate a model during fine tuning is very useful. It is also good to be able to customize the prompt, since it looks like CoT or other techniques can improve evaluation outputs.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the haystack-deepeval integration and reviewing how it currently selects GPT models and builds evaluation prompts. Compare that flow with deepeval's local Ollama usage; done means a local model and customized prompts can be used during RAG evaluation, with the supported configuration documented or covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ollama, python
- Domain
- ai, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100