confident-ai / confident-ai/deepeval
Issue with Component-level eval
- Dominant language
- Python
- Stars
- 18.3k
- Forks
- 1.9k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 19
Description
This code:
```
model = GeminiModel(
model_name="gemini-2.0-flash",
project=GCP_PROJECT_ID,
location=GCP_LOCATION,
temperature=0
)
@observe( metrics=[ContextualRelevancyMetric(model=model)])
def rag(query):
context = ["Dogs are animals"]
test_case = LLMTestCase(input=query, actual_output="I don't know :(", retrieval_context=context)
update_current_span(test_case=test_case)
return context
if __name__ == "__main__":
eval_results = evaluate(
goldens= [Golden(input="What are cats?")],
observed_callback= rag
)
print(eval_results)
```
got me that evaluation result (note that it didn't updated retrieval_context or actual_output). I think tracing isn't working well. Am i forgetting something?

Contributor guide
Research direction
Start by reproducing the provided GeminiModel, ContextualRelevancyMetric, observe, update_current_span, and evaluate example to inspect whether retrieval_context and actual_output reach the evaluation result. No files or tests are named; the issue is resolved when the traced fields are reflected correctly in component-level evaluation output.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100