Risk assessment model: turn attack results into a defensible leakage risk statement
- Vorherrschende Sprache
- Python
- Sterne
- 23
- Forks
- 28
- Ø Merge
- 4 T. 12 Std.
- Gemergte PRs (30 T.)
- 5
Beschreibung
# Feature
## Desired Behavior / Functionality
LeakPro measures how well attacks succeed, but it has no defensible way to turn that into a risk statement. What exists today is arbitrary:
1. The webapp scores risk as hardcoded AUC bands (`riskLevel()` in `leakpro/webapp/frontend/src/components/results/Summary.tsx`): `>= 0.75` HIGH, `>= 0.60` MEDIUM. The thresholds have no provenance, and AUC is an average-case measure that hides the worst-case per-record leakage which actually determines exposure. A model can sit at AUC 0.58 and still have training records that are perfectly identifiable.
2. The core library has no risk concept at all, so CLI and notebook users get nothing. `MIAResult` stops at `fpr`/`tpr`/`roc_auc`/`signal_values` plus the fixed-FPR table.
Proposal: a `leakpro/risk/` package that turns MIA results plus a declared use-case profile into a structured risk assessment, reported as a decomposition rather than a single opaque index.
**Measured** (from the audit, reproducible): TPR at a user-chosen FPR alpha, advantage `TPR(alpha) - alpha`, lift `TPR(alpha)/alpha`, the count and identity of members flagged at that operating point, generalization gap and DP-SGD epsilon as context.
**Declared** (by the data controller, echoed in the output): attacker prior `pi`, number of subjects, records per subject, severity, subject vulnerability, and an optional cost per exposed subject.
**Combined** (every factor traceable to one of the two blocks): PPV `pi*TPR / (pi*TPR + (1-pi)*alpha)`, expected exposed subjects, expected cost when a cost was supplied, and a band drawn from an explicit, versioned, user-overridable policy table that is always printed next to the band.
The reason PPV matters: audits run on a balanced member/non-member split, so every TPR and AUC LeakPro reports implicitly assumes `pi = 0.5`. At `TPR(1% FPR) = 0.10`, PPV is 91% at `pi = 0.5` but 1% at `pi = 0.001`. Reporting only the balanced figure overstates real-world exposure by an order of magnitude; reporting only a pessimistic one buries the memorization finding. Both belong in the output with `pi` visibly declared.
Deliberately excluded: no invented weights. Severity and subject vulnerability select a row in a sourced policy table, they do not multiply into a score.
## What Needs to Be Done
- `leakpro/risk/` with `schemas.py` (`UseCaseProfile`, `BandPolicy`, `RiskAssessment` as pydantic models, `extra="forbid"`), `vulnerability.py`, `assessment.py`, `policy.py`, `render.py`.
- Optional `use_case:` block on `LeakProConfig` for reproducibility. Parsing it does not trigger anything.
- Public API kept separate from the attack pipeline: `run_audit()` returns results, the user then calls `assess_risk(results, profile, num_train=...)`. No change to `run_audit`'s signature or return type.
- Optional `risk_assessment` parameter on `ReportHandler` for a Risk section in the PDF.
- Webapp: replace `riskLevel()` with a backend endpoint that calls the library, so no scoring lives in TypeScript.
- v1 covers MIA only, behind a `VulnerabilityMeasurement` protocol so MINV/GIA/synthetic can implement it later without reworking the risk layer.
Four guards are required, because each one is a way the naive version produces a confidently wrong number:
- TPR at a fixed FPR is `0.0` when no ROC point satisfies `fpr <= alpha` (`MIAResult._get_result_fixed_fpr` uses `max(..., default=0.0)`). A small audit set asked for `alpha = 0.001` therefore reports TPR 0, which reads as "no risk" for an operating point that was never measurable. The risk layer must refuse to report below `1 / n_non_members`.
- `attack_p` currently reports an inverted ROC for every target (measured AUC 0.214 where the correctly oriented value is 0.786). Any aggregation including it turns a real leak into a low-risk reading, so results with `auc < 0.5` must be flagged rather than averaged in. Separate bug, will be filed on its own.
- `MIAResult.get_strongest` maximizes `roc_auc`, but the attack relevant to risk is the one that wins at the user's operating point. These routinely disagree.
- Two TPR conventions coexist in the codebase: the live `MIAResult` (`leakpro/reporting/mia_result.py`) tabulates `fixed_fpr_table` as **fractions** with keys like `TPR@1%FPR`, while the legacy `leakpro/metrics/attack_result.py::find_tpr_at_fpr` returns **percentages** with keys like `TPR@1.0%FPR`. The risk layer accepts fractions only and rejects a percentage-valued table rather than letting a 100x unit error propagate into a risk figure.
**Correction to an earlier revision of this issue:** it claimed the webapp displays TPR@0.1%FPR as `1030.00%`. That is wrong and has been removed. The webapp reads the live `MIAResult.fixed_fpr_table`, which holds fractions, so its `value * 100` rendering is correct. The percentage convention belongs to the legacy `metrics/attack_result.py` helper, which nothing in the MIA path imports.
## How Can It Be Tested
- Unit tests in `leakpro/tests/risk/` mirroring the package. PPV checked against hand-computed values at `pi` in {0.5, 0.01, 0.001}; unresolvable-alpha case asserts a flag rather than TPR 0; an `auc = 0.214` result is rejected by name; attack selection picks the TPR winner over the AUC winner; a fixed-FPR value of `10.3` normalizes to `0.103` exactly once; a profile without `tolerated_fpr` is rejected; a profile with no cost yields no monetary figure rather than zero; identical inputs produce byte-identical output.
- End-to-end on an existing example (`examples/mia/gdd_ens` or `examples/mia/cifar`): run the audit, call `assess_risk`, confirm the saved JSON carries all three blocks plus the assumption list and policy version, and that the PDF compiles with the Risk section.
- Regression check that a config with no `use_case:` block behaves exactly as before.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.