Azure / Azure/azure-sdk-for-python
AdversarialSimulator should allow caller to specify a particular category of harm for generation
- Dominant language
- Python
- Stars
- 5.6k
- Forks
- 3.4k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 193
Description
**Is your feature request related to a problem? Please describe.**
My team uses the Azure AI simulator APIs to test the behavior of our generative AI system against harmful inputs. The simulator APIs, best as I can tell, do not give callers the ability to specify the category of harm that it is producing data for.
**Describe the solution you'd like**
The APIs should take additional parameters that allow callers to generate inputs that all map to a specific category of harm, e.g., only violent content, only self-harm content, etc.- or any combination thereof.
**Describe alternatives you've considered**
The only way to generate enough volume of a specific harm is to call the simulator sufficiently many times- but that results in a lot of unnecessary harmful data being generated which we have to then throw away. It'd be much nicer to have the finer-grained control via the API.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Assessment
This issue has not been assessed yet.