microsoft / microsoft/PyRIT

Wire doc/scanner notebooks into the integration notebook harness

Open
#1,752 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

not ready yet
Dominant language
Python
Stars
4.5k
Forks
893
Avg merge
3d 50m
Merged PRs (30d)
165

Description

## Problem

The `doc/scanner/` notebooks (`1_pyrit_scan.ipynb`, `airt.ipynb`, `benchmark.ipynb`, `foundry.ipynb`, `garak.ipynb`) are listed in `doc/myst.yml` and render in our docs site, but nothing in CI actually executes them. The integration notebook harness (`tests/integration//test_notebooks_*.py`) is hard-coded to `doc/code//` via `pyrit.common.path.DOCS_CODE_PATH`, so the scanner notebooks are render-only.

This means breakages in the user-facing scanner API surface — like the deprecated-type slip-through fixed in #1746 — won't be caught by CI.

## Proposal

Add `tests/integration/scanner/test_notebooks_scanner.py`, mirroring the existing per-area pattern (e.g. `tests/integration/scenarios/test_notebooks_scenarios.py`). It would parametrize over `os.listdir(DOC_ROOT / "scanner")` and run each notebook via `ExecutePreprocessor` under `RUN_ALL_TESTS=true`.

Points to confirm during implementation:
- The scanner notebooks call `pyrit_scan` end-to-end with `OpenAIChatTarget()` against real datasets. They are likely slow enough to warrant `max_dataset_size=1` shims or a `skipped_files` entry for the heavy ones — pick the right cost/coverage tradeoff up front.
- `DOCS_CODE_PATH` is `doc/code` by name; introducing a parallel `DOCS_SCANNER_PATH` constant in `pyrit/common/path.py` keeps the pattern consistent.

## Context

Surfaced in the PR review thread on #1746: https://github.com/microsoft/PyRIT/pull/1746#issuecomment-4478807681

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/integration/scenarios/test_notebooks_scenarios.py, pyrit.common.path.DOCS_CODE_PATH, and the notebooks under doc/scanner/. Compare the existing per-area harness and determine how max_dataset_size=1 or skipped_files should handle the slow notebooks. Done means the scanner notebooks execute under RUN_ALL_TESTS=true and their user-facing API paths are covered by CI.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.