facebookresearch / facebookresearch/ResponsibleNLP
Toxigen Metric not returning anything
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 212
- Forks
- 35
- PR merge metrics
- No merged PRs in 30d
Description
Hi! I'm trying to analyse some of my model's generations using the regard promptset. When I try to apply the regard metric everything seems to work fine, but when I try to use the toxigen metric it doesn't return anything. Here's the command I'm using:
robbie/scripts/run_eval.sh --dataset regard --model ../my_model -s toxigen --result-dir results/ --dataset-dir robbie/datasets/regard/
And this is the output:
MetricResult(scores=[], stats={})
Any help would be greatly appreciated. Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with robbie/scripts/run_eval.sh and trace how the toxigen metric is selected and executed, comparing that path with the working regard metric. Reproduce the command from the issue and inspect the metric result; done means the toxigen evaluation returns populated scores and stats instead of MetricResult(scores=[], stats={}).
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100