spring-projects / spring-projects/spring-ai

Expose EvaluationResponse from LLM

Open
#6,322 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-triage
Dominant language
Java
Stars
9.5k
Forks
2.9k
Avg merge
1d 7h
Merged PRs (30d)
6

Description

Expected Behavior

The Fact Checking Evaluator and the Relevancy Evaluator should also give the reasoning back. This is required for the testers to understand why the test is a pass. This gives transparency and also supports audits to be recorded as to why the score was pass one day and failed the other.

Current Behavior

The current evaluation framework also gives me a score and a success/fail. While this serves the purpose but creates confusion as to why its a pass or a fail. I tried passing my own prompt but more times the change of prompt made the yes.ignorecase() to fail and to pass me a fail, though the LLM was passing the reasoning as a pass.

Context

Because of the current behaviour I could not use the evaluators out of the box. As a workaround for now I created custom evaluators. However I would like to contribute to enrich the current code to satisfy these kind of extra cases.

Here is how I want to do it

  1. There is already a feedback that is now hardcoded to an empty string. I would like to use that field to pass back the reasoning by giving a way for the users to pass their custom prompt to wrap the reasoning as feedback
  2. Also my idea would be to expose the whole LLM response in the evaluation response so that users can write their custom prompts and can use the response in the way they want if they do not want to use the out of box score and success/fail.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the evaluation framework at the Fact Checking and Relevancy Evaluator entry points, then trace how EvaluationResponse is assembled and where feedback is hardcoded to an empty string. Determine how custom prompts and the complete LLM response should be represented; done means evaluators can return reasoning and expose the LLM response without losing the existing score and success/fail results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
ai, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.