aws-samples / aws-samples/foundation-model-benchmarking-tool
add hallucination metric to model evals on FMBench
- Dominant language
- Jupyter Notebook
- Stars
- 256
- Forks
- 44
- PR merge metrics
- No merged PRs in 30d
Description
FMBench can now evaluate models using a panel of LLM judges and give accuracy scores as to which candidate model is the most accurate. This issue is to do as follows:
1. Calculate a hallucination metric (that measures the amount of times a given response was actually a hallucination and incorrect)
2. Calculate the correctly "incorrect" answers, a.k.a number of times a candidate model said "i don't know" to a question rather than hallucinating and giving a response.
Contributor guide
Research direction
The issue identifies FMBench model evaluations and an existing panel of LLM judges with accuracy scores, but names no files, tests, or entry points. Start by tracing where those scores are computed; done means reporting hallucination counts and correctly "incorrect" ("I don't know") counts for each candidate model.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100