[AI Evaluation] Failed to parse score for 'Groundedness' from the following evaluation response:
- Dominant language
- C#
- Stars
- 3.2k
- Forks
- 894
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
### Description
Hi there! I'm reaching a similar error through Ollama (local model), Gemini, and via Amazon Bedrock, also it can be found by using `CompositeEvaluator` or directly by `GroundednessEvaluator` (and others)
Note: Via Gemini and Amazon Bedrock I'm using Semantic Kernel's connectors.
The error appears into the `Diagnostics` list.
## Ollama
Here is an example just using `Microsoft.Extensions.AI.*`, with local Ollama: [Microsoft.Extensions.AI.Evaluation.Tests.Ollama](https://github.com/henriqueholtz/Microsoft.Extensions.AI.Evaluation.Tests/tree/master/Microsoft.Extensions.AI.Evaluation.Tests.Ollama)
Ollama Error details
```
Expected evaluationMetric.Interpretation?.Rating to be one of {EvaluationRating.Good {value: 5}, EvaluationRating.Exceptional {value: 6}}
because -------------------------------------
Failed: False
Reason:
Interpretation Reason:
Interpretation Rating: Inconclusive
Diagnostics Count: 1: Failed to parse score for 'Groundedness' from the following evaluation response:
Let's think step by step:
1. The CONTEXT provides information about the order ID (123) and the tracking code (TKG_ABC).
2. The QUERY is a direct question about the tracking for the order 123.
3. The RESPONSE directly answers the query by providing the tracking information for the order 123.
Explanation: The response is completely relevant to the context and query, providing the exact information requested. Therefore, the score should be [Groundedness: 5].
Score: 5
-------------------------------------
Query: What is the tracking for the order 123?
-------------------------------------
ChatResponse: OrderId is 123, Tracking code is TKG_ABC.
, but found EvaluationRating.Inconclusive {value: 1}.
at AwesomeAssertions.Execution.LateBoundTestFramework.Throw(String message)
at AwesomeAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
at AwesomeAssertions.Execution.AssertionScope.AddPreFormattedFailure(String formattedFailureMessage)
at AwesomeAssertions.Execution.AssertionChain.FailWith(Func`1 getFailureReason)
at AwesomeAssertions.Execution.AssertionChain.FailWith(Func`1 getFailureReason)
at AwesomeAssertions.Execution.AssertionChain.FailWith(String message, Object[] args)
at AwesomeAssertions.Primitives.EnumAssertions`2.BeOneOf(IEnumerable`1 validValues, String because, Object[] becauseArgs)
at Microsoft.Extensions.AI.Evaluation.Tests.Ollama.CompositeEvaluatorTests.CompositeEvaluatorWithGroundednessEvaluatorTest() in D:\Repositories\Microsoft.Extensions.AI.Evaluation.Tests\Microsoft.Extensions.AI.Evaluation.Tests.Ollama\CompositeEvaluatorTests.cs:line 55
--- End of stack trace from previous location ---
```
## Gemini
Here is an example using `Microsoft.Extensions.AI.*` + `Microsoft.SemanticKernel.Connectors.Google` (which is currently in alpha version) with Gemini: [Microsoft.Extensions.AI.Evaluation.Tests.Gemini](https://github.com/henriqueholtz/Microsoft.Extensions.AI.Evaluation.Tests/tree/master/Microsoft.Extensions.AI.Evaluation.Tests.Gemini) - which is by default using `gemini-2.5-pro`.
Note: Not sure if the problem is coming from Semantic Kernel's connector or from `Microsoft.Extensions.AI.Evaluation.*`
Gemini Error Details
```
Expected evaluationMetric.Interpretation?.Rating to be one of {EvaluationRating.Good {value: 5}, EvaluationRating.Exceptional {value: 6}}
because -------------------------------------
Failed: False
Reason:
Interpretation Reason:
Interpretation Rating: Inconclusive
Diagnostics Count: 1: Failed to parse score for 'Groundedness' from the following evaluation response:
Let's think step by step:
1. **Analyze the Query:** The user wants to know the tracking code for a specific order, "order 123".
2. **Analyze the Context:** The context provides two pieces of information: "OrderId is 123"
-------------------------------------
Query: What is the tracking for the order 123?
-------------------------------------
ChatResponse: OrderId is 123, Tracking code is TKG_ABC.
, but found EvaluationRating.Inconclusive {value: 1}.
at AwesomeAssertions.Execution.LateBoundTestFramework.Throw(String message)
at AwesomeAssertions.Execution.DefaultAssertionStrategy.HandleFailure(String message)
at AwesomeAssertions.Execution.AssertionScope.AddPreFormattedFailure(String formattedFailureMessage)
at AwesomeAssertions.Execution.AssertionChain.FailWith(Func`1 getFailureReason)
at AwesomeAssertions.Execution.AssertionChain.FailWith(Func`1 getFailureReason)
at AwesomeAssertions.Execution.AssertionChain.FailWith(String message, Object[] args)
at AwesomeAssertions.Primitives.EnumAssertions`2.BeOneOf(IEnumerable`1 validValues, String because, Object[] becauseArgs)
at Microsoft.Extensions.AI.Evaluation.Tests.Gemini.CompositeEvaluatorTests.CompositeEvaluatorWithGroundednessEvaluatorTest() in D:\Repositories\Microsoft.Extensions.AI.Evaluation.Tests\Microsoft.Extensions.AI.Evaluation.Tests.Gemini\CompositeEvaluatorTests.cs:line 75
--- End of stack trace from previous location ---
```
---
### Reproduction Steps
#### Ollama
As mentioned in the [Ollama README](https://github.com/henriqueholtz/Microsoft.Extensions.AI.Evaluation.Tests/blob/master/Microsoft.Extensions.AI.Evaluation.Tests.Ollama/README.md):
- Run the ollama through docker: `docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama`
- Pull the llama2 model: `docker exec -it ollama ollama pull llama2`
- Run the tests
#### Gemini
As mentioned in the [Gemini README](https://github.com/henriqueholtz/Microsoft.Extensions.AI.Evaluation.Tests/blob/master/Microsoft.Extensions.AI.Evaluation.Tests.Gemini/README.md)
1. `dotnet user-secrets init --project ./Microsoft.Extensions.AI.Evaluation.Tests.Gemini/Microsoft.Extensions.AI.Evaluation.Tests.Gemini.csproj`
2. `dotnet user-secrets set "GeminiApiKey" "" --project ./Microsoft.Extensions.AI.Evaluation.Tests.Gemini/Microsoft.Extensions.AI.Evaluation.Tests.Gemini.csproj`
3. Run the tests
### Expected behavior
The score, interpretation etc should be parsed correctly
### Actual behavior
The score cannot be parsed
### Regression?
_No response_
### Known Workarounds
_No response_
### Configuration
- Windows
- .NET 9 SDK
### Other information
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.