spring-projects / spring-projects/spring-ai
RelevancyEvaluator problem
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 9.5k
- Forks
- 2.9k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 6
Description
Bug description
method evaluate of RelevancyEvaluator always returns passing false
Problem is with Git commit caed4b3705126ed8dfd2077c3177c9c7c2b8a963 of @remeio
https://github.com/spring-projects/spring-ai/commit/caed4b3705126ed8dfd2077c3177c9c7c2b8a963
specifically with this condition:
if ("yes".equalsIgnoreCase(evaluationResponse))
Because LLM response is "YES\n", due to DEFAULT_PROMPT_TEMPLATE:
"Answer YES, if the response for the query is in line with context information otherwise NO."
And the condition fails because of the line break after "YES" (because the using of "equals" instead of "contains")
Environment
Spring AI version 1.1.0
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with RelevancyEvaluator.evaluate and the referenced commit caed4b3705126ed8dfd2077c3177c9c7c2b8a963. Reproduce the issue with the LLM response "YES\n" and trace how DEFAULT_PROMPT_TEMPLATE leads to that response. Done means a newline-terminated affirmative response is recognized as passing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- ai
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100