Owner

confident-ai

4 indexed repositories · View on GitHub

deepeval
The LLM Evaluation Framework

Python · 18271 stars
deepteam
DeepTeam is a framework to red team LLMs and AI agents.

Python · 2803 stars
blog-examples

Python · 14 stars
confident-agent

Java · 0 stars

Open All

Module 'openai' has no attribute 'Embedding' Open

confident-ai/blog-examples#1 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
A metrics class containing different definitions of metrics ordered by some taxonomy. Open

confident-ai/deepeval#238 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Metric Request: PII Score Open

confident-ai/deepeval#245 · 5 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Add JudgeLM as a way to evaluate and compare historical test runs Open

enhancement help wanted

confident-ai/deepeval#262 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Disk Error When Attempting to Run Tests Open

confident-ai/deepeval#286 · 34 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Faithfulness score and SummaC model parameters are not very much clear. Open

confident-ai/deepeval#310 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Integration of LM Evaluation Harness. Open

confident-ai/deepeval#332 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Add metric for NER Open

confident-ai/deepeval#352 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Make Own Implementation of Dbias and Detoxify Open

help wanted

confident-ai/deepeval#382 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
llama index integration broken at latest versions Open

confident-ai/deepeval#441 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
AttributeError: 'AIMessage' object has no attribute 'find' Open

confident-ai/deepeval#482 · 9 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
retrieval_context Open

confident-ai/deepeval#483 · 12 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
ValueError: Number of verdicts generated does not equal length of retrieved nodes. Open

confident-ai/deepeval#484 · 5 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Having so many random errors while scoring Open

confident-ai/deepeval#489 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Trainer needs some further definition in DeepEvalHuggingFaceCallback Open

confident-ai/deepeval#496 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Parallelization of Evaluations Open

confident-ai/deepeval#500 · 12 comments · 3 reactions · 0 assignees · Indexed Sep 15, 2026
Support popular benchmarks Open

enhancement help wanted

confident-ai/deepeval#508 · 11 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Evaluation Dataset Generation Open

enhancement help wanted

confident-ai/deepeval#530 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Make multithreading optional Open

confident-ai/deepeval#538 · 8 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Summarization metric generates <1 score for exactly the same input and output Open

confident-ai/deepeval#565 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Custom dataset evaluation giving ValueError: Input, actual output, and retrieval context cannot be None Open

confident-ai/deepeval#568 · 13 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Store computed metrics in a dataframe Open

confident-ai/deepeval#595 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Consideration behind a "stateful" metric UX Open

confident-ai/deepeval#597 · 9 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
temp_test_run.json contains partial entries when running in bulk Open

confident-ai/deepeval#605 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Add browser opening flag to config Open

confident-ai/deepeval#645 · 1 comment · 2 reactions · 0 assignees · Indexed Sep 15, 2026
Writing of any files like cache or temp to running directory - Disable or Provide Config to set location Open

confident-ai/deepeval#671 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Support batching for custom models Open

confident-ai/deepeval#702 · 9 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Hallucination test is really a test for faithful summarization, not of hallucination-absence Open

confident-ai/deepeval#730 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Show me the prompt! Open

confident-ai/deepeval#731 · 2 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026
Return type of `evaluate` is not documented. Open

confident-ai/deepeval#733 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Score is not cached Open

confident-ai/deepeval#734 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
No test cases found error Open

confident-ai/deepeval#739 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Cant use HuggingFace Model for evaluation Open

confident-ai/deepeval#743 · 16 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Unable to disable telemetry Open

confident-ai/deepeval#757 · 12 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
.generate_raw_response(prompt) gives an attribute error Open

confident-ai/deepeval#767 · 7 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Corrupted cache can make it impossible to run deepeval Open

confident-ai/deepeval#768 · 0 comments · 5 reactions · 0 assignees · Indexed Sep 15, 2026
Remove unconditional, uncontrollable output statements Open

confident-ai/deepeval#769 · 2 comments · 11 reactions · 0 assignees · Indexed Sep 15, 2026
Dramatically simpler and more reliable cache Open

confident-ai/deepeval#775 · 1 comment · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Suggestion for Future Improvement: Enable Function Calling for Consistent JSON Output Open

confident-ai/deepeval#790 · 0 comments · 8 reactions · 0 assignees · Indexed Sep 15, 2026
No response after processing few prompts. Open

confident-ai/deepeval#792 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
ProxySchemeUnknown: Proxy URL had no scheme, should start with http:// or https:// Open

confident-ai/deepeval#793 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
`check_for_update` should not happen on package load Open

confident-ai/deepeval#794 · 2 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026
accuracy always comes 0, might be a bug in my code i am unable to find. Open

confident-ai/deepeval#819 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Message about nest_asyncio Should Not Be Printed Open

confident-ai/deepeval#821 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Add Support for `Gemini` Models Open

confident-ai/deepeval#829 · 7 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Conversation Evaluations Open

confident-ai/deepeval#836 · 0 comments · 3 reactions · 0 assignees · Indexed Sep 15, 2026
MMLU stopped working Open

confident-ai/deepeval#838 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Invalid value for 'top_logprobs': must be less than or equal to 5 for AzureOpenAi Models Open

confident-ai/deepeval#843 · 2 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
AuthenticationError Open

confident-ai/deepeval#844 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Update to tenacity 8.4.1 Open

confident-ai/deepeval#847 · 4 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026

Showing the newest 100

This page lists what was indexed most recently. Advanced filter has the whole inventory, narrowed by language, difficulty and how long a task takes.

Open advanced filter

confident-ai

Showing the newest 100

Get new issues in your inbox