aws-samples / aws-samples/foundation-model-benchmarking-tool

add hallucination metric to model evals on FMBench

Open
#170 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Jupyter Notebook
Stars
256
Forks
44
PR merge metrics
No merged PRs in 30d

Description

FMBench can now evaluate models using a panel of LLM judges and give accuracy scores as to which candidate model is the most accurate. This issue is to do as follows:

1. Calculate a hallucination metric (that measures the amount of times a given response was actually a hallucination and incorrect)
2. Calculate the correctly "incorrect" answers, a.k.a number of times a candidate model said "i don't know" to a question rather than hallucinating and giving a response.

Contributor guide

Open the contributing guide

Research direction

The issue identifies FMBench model evaluations and an existing panel of LLM judges with accuracy scores, but names no files, tests, or entry points. Start by tracing where those scores are computed; done means reporting hallucination counts and correctly "incorrect" ("I don't know") counts for each candidate model.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.