spring-projects / spring-projects/spring-ai
Expose EvaluationResponse from LLM
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 6
Description
Expected Behavior
The Fact Checking Evaluator and the Relevancy Evaluator should also give the reasoning back. This is required for the testers to understand why the test is a pass. This gives transparency and also supports audits to be recorded as to why the score was pass one day and failed the other.
Current Behavior
The current evaluation framework also gives me a score and a success/fail. While this serves the purpose but creates confusion as to why its a pass or a fail. I tried passing my own prompt but more times the change of prompt made the yes.ignorecase() to fail and to pass me a fail, though the LLM was passing the reasoning as a pass.
Context
Because of the current behaviour I could not use the evaluators out of the box. As a workaround for now I created custom evaluators. However I would like to contribute to enrich the current code to satisfy these kind of extra cases.
Here is how I want to do it
- There is already a feedback that is now hardcoded to an empty string. I would like to use that field to pass back the reasoning by giving a way for the users to pass their custom prompt to wrap the reasoning as feedback
- Also my idea would be to expose the whole LLM response in the evaluation response so that users can write their custom prompts and can use the response in the way they want if they do not want to use the out of box score and success/fail.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in the evaluation framework at the Fact Checking and Relevancy Evaluator entry points, then trace how EvaluationResponse is assembled and where feedback is hardcoded to an empty string. Determine how custom prompts and the complete LLM response should be represented; done means evaluators can return reasoning and expose the LLM response without losing the existing score and success/fail results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring
- Domain
- ai, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100