microsoft / microsoft/PyRIT

Parent: LLM Vulnerability Scanner Feature Parity with Garak, Giskard, CyberSecEval

Abierto
#511 2 comentarios 2 reacciones 0 asignados Ver en GitHub
good first issue help wanted
Lenguaje dominante
Python
Estrellas
4.5k
Forks
893
Merge medio
3 d 50 min
PR fusionados (30 d)
165

Descripción

TLDR; We want help creating issues where Garak, Giskard, or CyberSecEval support an attack scenario where PyRIT doesn't support it.

#### Background

Other tools like Garak, Giskard, and CyberSecEval have different use cases from PyRIT, but there is overlap as well. If there is an attack in one of these other platforms, we probably want to support it also and would love issues being created so we can track this (or clarify if it already exists/improve our docs).

[This](https://arxiv.org/abs/2410.16527) paper tackles a really difficult problem, and that's comparing these tools (and PyRIT). When I was looking, some of the things I noticed that it mentioned PyRIT was lacking were:

- Lack of multi-language support (I think we actually have this as a converter)
- Lack of GCG (we have this as an auxiliary module)
- Dan, AutoDan, etc (I think we have support for all of this. We certainly have DAN and ways to modify those prompts with GPTFuzz - but we would love to know what we're missing)
- Insecure Coding Test (we don't have this, but it would be an easy scorer to write!)

And there might be more that I missed!

So as an example of issues we'd like, all of the above are good to open. Even if we think we support them (like in the multi-language case) we can dive into specifics to make sure we're not missing anything, and potentially improve documentation so it's more obvious these scenarios are supported.

#### Describe the solution you'd like

Please open follow-up sub-issues with any features you notice or find in these other tools that PyRIT doesn't have! That way we can track them. Also, we would love community to tackle any of these also, but it still can be helpful to open an issue so we can help guide :D

Guía de contribución

No hay ninguna guía de contribución indexada para este repositorio

Línea de trabajo

No se identifica ningún archivo de implementación, prueba ni punto de entrada. Empieza comparando los escenarios de ataque compatibles con Garak, Giskard y CyberSecEval con las capacidades documentadas de PyRIT y, después, abre una issue de seguimiento centrada en un escenario que falte; se considerará terminado cuando el escenario tenga un alcance claro de implementación o documentación.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
python
Área
security
Tipo de issue
Nueva funcionalidad
Dificultad
5/5
Tiempo estimado
Más de una semana
Estado de actividad
Tranquilo
Claridad
Necesita aclaración
Aptitud para principiantes
25/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.