microsoft / microsoft/PyRIT

FEAT: Add HarmActionsEval for evaluation of AI agent action safety

Open
#1,553 3 comments 0 reactions 0 assignees View on GitHub
not ready yet
Dominant language
Python
Stars
4.5k
Forks
893
Avg merge
3d 50m
Merged PRs (30d)
165

Description

## Background

AI agents have a growing adoption across the industry, including critical applications. AI agents that have access to tools can currently call tools directly with no centralized validation layer that inspects these calls before execution, allowing harmful or disallowed tool calls to be executed without oversight.
Existing agent benchmarks evaluate the safety of final responses instead of actions.

HarmActionsEval benchmark evaluates actions. It found that 80% of the LLMs tested executed actions at the first attempt for over 95% of the harmful prompts.

Related work: https://github.com/Pro-GenAI/Agent-Action-Guard.

## Proposed change

To integrate HarmActionsEval in this project.

If the maintainers express interest, I will write the code and create a Pull Request.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are identified in the issue. Start by reviewing PyRIT’s existing evaluation capabilities and the related Agent-Action-Guard project, then clarify the integration scope and acceptance criteria with maintainers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.