FEAT: Add HarmActionsEval for evaluation of AI agent action safety
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 893
- Avg merge
- 3d 50m
- Merged PRs (30d)
- 165
Description
## Background
AI agents have a growing adoption across the industry, including critical applications. AI agents that have access to tools can currently call tools directly with no centralized validation layer that inspects these calls before execution, allowing harmful or disallowed tool calls to be executed without oversight.
Existing agent benchmarks evaluate the safety of final responses instead of actions.
HarmActionsEval benchmark evaluates actions. It found that 80% of the LLMs tested executed actions at the first attempt for over 95% of the harmful prompts.
Related work: https://github.com/Pro-GenAI/Agent-Action-Guard.
## Proposed change
To integrate HarmActionsEval in this project.
If the maintainers express interest, I will write the code and create a Pull Request.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files, tests, or entry points are identified in the issue. Start by reviewing PyRIT’s existing evaluation capabilities and the related Agent-Action-Guard project, then clarify the integration scope and acceptance criteria with maintainers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100