confident-ai / confident-ai/deepeval

Issue with Component-level eval

Open
#1,663 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
18.3k
Forks
1.9k
Avg merge
2d 6h
Merged PRs (30d)
19

Description

This code:

```
model = GeminiModel(
model_name="gemini-2.0-flash",
project=GCP_PROJECT_ID,
location=GCP_LOCATION,
temperature=0
)

@observe( metrics=[ContextualRelevancyMetric(model=model)])
def rag(query):
context = ["Dogs are animals"]
test_case = LLMTestCase(input=query, actual_output="I don't know :(", retrieval_context=context)
update_current_span(test_case=test_case)
return context

if __name__ == "__main__":
eval_results = evaluate(
goldens= [Golden(input="What are cats?")],
observed_callback= rag
)
print(eval_results)
```

got me that evaluation result (note that it didn't updated retrieval_context or actual_output). I think tracing isn't working well. Am i forgetting something?

![Image](https://github.com/user-attachments/assets/0a6d7afb-3581-429b-9c85-94578e48816f)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the provided GeminiModel, ContextualRelevancyMetric, observe, update_current_span, and evaluate example to inspect whether retrieval_context and actual_output reach the evaluation result. No files or tests are named; the issue is resolved when the traced fields are reflected correctly in component-level evaluation output.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.