Azure / Azure/azure-sdk-for-python

AdversarialSimulator should allow caller to specify a particular category of harm for generation

Open
#40,749 3 comments 0 reactions 0 assignees View on GitHub
Client customer-reported Evaluation feature-request needs-team-attention Service Attention
Dominant language
Python
Stars
5.6k
Forks
3.4k
Avg merge
1d 21h
Merged PRs (30d)
193

Description

**Is your feature request related to a problem? Please describe.**
My team uses the Azure AI simulator APIs to test the behavior of our generative AI system against harmful inputs. The simulator APIs, best as I can tell, do not give callers the ability to specify the category of harm that it is producing data for.

**Describe the solution you'd like**
The APIs should take additional parameters that allow callers to generate inputs that all map to a specific category of harm, e.g., only violent content, only self-harm content, etc.- or any combination thereof.

**Describe alternatives you've considered**
The only way to generate enough volume of a specific harm is to call the simulator sufficiently many times- but that results in a lot of unnecessary harmful data being generated which we have to then throw away. It'd be much nicer to have the finer-grained control via the API.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.