NVIDIA / NVIDIA/garak

Migrate owasp:llmNN probe tags to year-qualified owasp:YYYY:llmNN format

Open
#2,139 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
9.3k
Forks
1.3k
Avg merge
5d 1h
Merged PRs (30d)
14

Description

Spun out of #1919 and #1920 per @jmartin-tech's review.

Background

agent_breaker.AgentBreaker is currently the only probe tagged owasp:llm07 / owasp:llm08, while its doc_uri points at LLM06 (Excessive Agency) in the 2025 list. #1920 attempted to retag just that one probe, but as noted in review that would leave main inconsistent: the owasp:llm06 tag in garak was defined against the 2023-24 OWASP Top 10, tag-based grouping selects and groups by prefix, and changing a single probe conflates taxonomy versions.

Proposal

Migrate every owasp:llmNN tag in the probe catalog (78 entries) to a year-qualified form, e.g. for AgentBreaker:

owasp:2023:llm07
owasp:2023:llm08
owasp:2025:llm06
owasp:2026:llm03
  • At minimum, add the 2023 year marker to all existing entries so their meaning is unchanged.
  • Add newer-list tags (2025, 2026) where a probe maps onto them.
  • Update tag selection and grouping logic that keys on the owasp:llm prefix so owasp:2023:*, owasp:2025:* etc. group correctly.
  • Update docs that reference the prefix scheme.

Open questions

  • Preferred spelling for the year-qualified prefix (owasp:2023:llm07 vs. an alternative).
  • Whether newer-list tags (2025 / 2026) land in the same PR or a follow-up.

Happy to own this PR if the team wants to proceed.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the probe catalog entries described here, then trace the tag selection and grouping logic that currently keys on the owasp:llm prefix. Check the context in issues #1919 and #1920 before choosing the year-prefix spelling and scope for newer-list tags. Done means existing tags are year-qualified, applicable newer tags are handled, grouping remains correct, and referenced documentation is updated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, security, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.