deepset-ai / deepset-ai/haystack

Handle errors separately in evaluators and the run results

Open
#7,973 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

shaping topic:eval
Dominant language
Python
Stars
26.6k
Forks
3.2k
Avg merge
1d 3h
Merged PRs (30d)
194

Description

Context

When running the evaluators over larger datasets, depending on the model, it is very common to run into LLM errors where the output is not valid JSON. For example, while running the benchmark scripts over the ARAGOG dataset, I always have one row that got incorrect JSON, so every time I've run the script I get a score report which is not very useful, such as:

metrics score
context_relevance NaN

In that case, the output of the LLM-based evaluation metric whenever there is an error is something like:
{'statements': [], 'statement_scores': [], 'score': nan}

As a user, I would like to keep track of the errors that happened during evaluation, so ideally this should be returned as a flag, for example:
{'statements': [], 'statement_scores': [], 'score': nan, 'error': True}

Then, in the evaluation score report, we could return the mean of the scores by ignoring the errors:

metrics score total_errors
context_relevance 0.9 1

Outcome

  • Changes to the LLM-based evaluators (context relevancy and faithfulness) so they return an error Flag.
  • Changes to the LLM-based evaluators to return a score even if there are rows with np.nan (for example, a suggestion is to change np.mean with np.nanmean here).
  • Changes to the score_report() function of the EvaluationRunResult to return total errors.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the context_relevance evaluator at haystack/components/evaluators/context_relevance.py#L179 and compare its handling of invalid JSON with the faithfulness evaluator. Then trace EvaluationRunResult.score_report() to define how evaluator errors and NaN scores are represented. Done means both LLM-based evaluators expose errors and score_report() reports scores while returning total errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.