FEAT Add Anthropic/model-written-evals Dataset
@lxcxjxhx is already working on this.
Since Jul 19, 2026.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 893
- Avg merge
- 3d 50m
- Merged PRs (30d)
- 165
Description
> Name: Anthropic/model-written-evals
>
> Link: https://huggingface.co/datasets/Anthropic/model-written-evals
>
> Relevant columns: "question", "answer_matching_behavior"
_Originally posted by @divyaamin9825 in [#429](https://github.com/Azure/PyRIT/issues/429#issuecomment-2394713228)_
---
### Describe the solution you'd like
This dataset should be available within PyRIT: https://huggingface.co/datasets/Anthropic/model-written-evals
Also available here: https://github.com/anthropics/evals
Associated paper: https://arxiv.org/abs/2212.09251
### Additional context
There are examples of how PyRIT interacts with other datasets here: https://github.com/search?q=repo%3AAzure%2FPyRIT%20%23%20The%20dataset%20sources%20can%20be%20found%20at%3A&type=code
**_[[Content Warning: Prompts are aimed at provoking the model, and may contain offensive content.]]_**
_Additional Disclaimer: Given the content of these prompts, keep in mind that you may want to check with your relevant legal department before trying them against LLMs._
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.