Wire doc/scanner notebooks into the integration notebook harness
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 4.5k
- Forks
- 893
- Avg merge
- 3d 50m
- Merged PRs (30d)
- 165
Description
## Problem
The `doc/scanner/` notebooks (`1_pyrit_scan.ipynb`, `airt.ipynb`, `benchmark.ipynb`, `foundry.ipynb`, `garak.ipynb`) are listed in `doc/myst.yml` and render in our docs site, but nothing in CI actually executes them. The integration notebook harness (`tests/integration//test_notebooks_*.py`) is hard-coded to `doc/code//` via `pyrit.common.path.DOCS_CODE_PATH`, so the scanner notebooks are render-only.
This means breakages in the user-facing scanner API surface — like the deprecated-type slip-through fixed in #1746 — won't be caught by CI.
## Proposal
Add `tests/integration/scanner/test_notebooks_scanner.py`, mirroring the existing per-area pattern (e.g. `tests/integration/scenarios/test_notebooks_scenarios.py`). It would parametrize over `os.listdir(DOC_ROOT / "scanner")` and run each notebook via `ExecutePreprocessor` under `RUN_ALL_TESTS=true`.
Points to confirm during implementation:
- The scanner notebooks call `pyrit_scan` end-to-end with `OpenAIChatTarget()` against real datasets. They are likely slow enough to warrant `max_dataset_size=1` shims or a `skipped_files` entry for the heavy ones — pick the right cost/coverage tradeoff up front.
- `DOCS_CODE_PATH` is `doc/code` by name; introducing a parallel `DOCS_SCANNER_PATH` constant in `pyrit/common/path.py` keeps the pattern consistent.
## Context
Surfaced in the PR review thread on #1746: https://github.com/microsoft/PyRIT/pull/1746#issuecomment-4478807681
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/integration/scenarios/test_notebooks_scenarios.py, pyrit.common.path.DOCS_CODE_PATH, and the notebooks under doc/scanner/. Compare the existing per-area harness and determine how max_dataset_size=1 or skipped_files should handle the slow notebooks. Done means the scanner notebooks execute under RUN_ALL_TESTS=true and their user-facing API paths are covered by CI.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100