Airlock: support multiple/pluggable scan types (malware, PII) via a first-class scan-result contract

Summary

Airlock currently supports a single content check — Microsoft Defender for Storage malware scanning — and its result is expressed through the existing step-result state machine, where the scan completion is reported as completed_step="submitted" (the scan is the gate on the Submitted → InReview transition). We should introduce a first-class scan-result concept and support multiple, pluggable scan types (e.g. malware and PII / sensitive-data classification), with the airlock gate aggregating their verdicts.

Motivation

• Overloaded contract. A scan verdict conceptually belongs to a request regardless of the request's state, but today it is squeezed into the AirlockRequestStatus-keyed step-result contract (completed_step="submitted"). There is no ScanStatus/ScanResult concept — only AirlockRequestStatus. This overload is what makes the v2 timing awkward (see below).
• v2 timing workaround. In the consolidated (v2) storage model the blob is scanned on the Draft upload, so the submitted-step result can arrive before the user submits. PR #5048 added an interim pendingScanResult store on the request (API records the verdict during Draft and applies it on submission) to avoid dead-lettering. That works, but it exists only because the scan result has to masquerade as a state transition.
• Only one scanner. There is no way to add additional checks (PII, data classification, custom scanners) — the gate is hard-wired to a single malware verdict.

Proposal

Model scans as their own results, decoupled from the request state machine:

1. ScanResult event / record. A scanner emits { request_id, scanner, verdict, details } (e.g. scanner="malware", verdict="clean" | "malicious"; scanner="pii", verdict="clean" | "found").
2. Record on the request. The API stores verdicts in a scanResults map on the airlock request, independent of the request's current state — so a Draft-time result is simply "scan X done for request Y", not a state mismatch. This subsumes the interim pendingScanResult field from #5048.
3. Gate consumes the map. The Submitted → InReview transition becomes a gate evaluated when a result lands (and the request is submitted) or at submission:
   • not all expected scanners reported → wait,
   • all present and all pass → InReview,
   • any hard-fail → Blocked (with a structured, per-scanner reason).
4. Config-driven expected set. Which scanners must run (malware always; PII when enabled, etc.) is configuration the gate reads from the same source that drives the scanners, so the gate never waits on a scan that never runs.

This removes the completed_step="submitted" overload (and the pendingScanResult special-case), and drops straight into supporting additional scan types.

Design considerations

• Event schema / versioning — new event type + data_version; migrate the malware path off the step-result overload.
• Advisory vs blocking per scanner — PII is often "route to review with findings surfaced to the reviewer" rather than a hard block; policy belongs next to the gate, per scanner.
• Where each scanner runs — malware is Defender on the storage account; PII would be a separate service (e.g. Presidio / Purview / a custom function) that normalises into the same ScanResult shape.
• Timeouts / missing scanners — a max-wait and a policy for a scanner that never reports (block vs proceed-with-warning), so a stuck scanner can't wedge a request.
• Structured reasons — statusMessage (or a dedicated field) becomes a per-scanner breakdown rather than a single string.
• UI / CLI — surface per-scanner verdicts/findings to reviewers.

PII scanning: pluggable detection engines

The pii scanner is a provider behind the ScanResult contract, so the detection engine can be swapped by configuration without affecting the gate. The airlock never talks to a specific engine directly — it talks to the scanner service, which normalises whatever engine it uses into the common { request_id, scanner: "pii", verdict, details } shape.

Candidate engines:

• Microsoft Presidio (recommended default). Open source (Apache 2.0), self-hostable as containers inside the TRE Core vnet, so airlock content never leaves the TRE security boundary — consistent with the whole point of a TRE and mirroring how Defender scans storage in-place. Supports custom recognizers (regex/NLP) for domain-specific research identifiers (study IDs, NHS numbers, participant schemes) and a structured/batch mode for column-wise analysis of tabular data.
• Azure AI Language – PII detection (opt-in alternative). SaaS/PaaS API, zero infra to run, Microsoft-maintained models and broad multi-language coverage out of the box. Trade-off: content is processed by a Microsoft-managed cognitive service, which cuts against TRE data-residency guarantees unless used via a private endpoint within the tenant. Custom entities require trained Custom NER models.
• Custom / other engines — Purview, a bespoke function, or any engine that can emit the normalised ScanResult shape.

Because all engines normalise to the same contract, engine choice is a deployment config decision, not an architectural fork. Per-engine config (endpoint URL + auth) is supplied to the shared scanner service.

PII scanning: recursive content extraction & coverage-aware verdicts

PII detection engines only analyse text, but PII can hide anywhere in an airlock request — inside archives, documents, images, structured data and nested content. The scanner is therefore really a recursive content-extraction pipeline in front of the detection engine, not just the engine itself. Extraction is the hard part; the engine is pluggable behind it.

blob
 ├─ type detection (magic bytes, not extension)
 ├─ if container/archive  → expand → recurse into each member
 ├─ if document (PDF/DOCX/XLSX) → extract text + embedded objects → recurse
 ├─ if image → OCR → text
 ├─ if structured (CSV/Parquet/JSON) → column-wise analysis
 └─ if text → analyse directly
        → PII engine (Presidio / Azure AI Language / custom) → normalise → contributes to ScanResult

Archives and nested content

• Nested archives (zip-in-zip, tar.gz, 7z, …) — recurse, but with a depth limit and total-expanded-size / file-count cap to prevent zip-bomb denial-of-service against the in-boundary service.
• Encrypted / password-protected archives — cannot be extracted → distinct verdict (not "clean").
• Embedded content — PII also hides in PDF attachments, OLE objects in Office docs, image EXIF/metadata and notebook output cells; recursion should follow those too.

Critical rule: "can't extract" ≠ "clean". The scanner must never silently pass content it did not actually inspect. Each leaf resolves to one of:

Extraction outcome	Verdict	Gate behaviour
Extracted & analysed, no PII	clean	pass
PII found	found	advisory → surface findings to reviewer (or block, per policy)
Encrypted / password-protected	unextractable	route to manual review
Unknown binary / no extractor	unsupported	route to manual review
Depth/size/bomb limit hit	incomplete	route to manual review

The whole-request PII verdict is the aggregate of its leaves: any leaf that could not be fully inspected forces human review rather than an automatic pass — fail-toward-review, never fail-toward-pass, on anything not fully extracted.

Additional notes

• Extraction is a first-class, swappable stage with archive/document/image/structured handlers, independent of the chosen detection engine.
• Structured mode for tabular data — prefer column-level analysis over flattening to text, giving per-column findings that are more useful to reviewers.
• Guardrails — max recursion depth, max expanded size, max file count, per-file timeout — all under the max-wait policy above so a large or hostile upload can't wedge a request.
• Rich details — report findings and coverage: which paths inside an archive were inspected, and which were unsupported/unextractable/incomplete, so reviewers see exactly what was and wasn't machine-checked.
• All in-boundary — extraction, OCR and the detection engine (when Presidio) run inside the TRE Core vnet, consistent with TRE data-residency guarantees.

References

• Discussed on PR #5048 (Redesign airlock storage account architecture), which added the interim pendingScanResult store for the v2 Draft-time scan timing.
• Related malware-scan issues: #4978, #4403, #4030.

Start by reading PR #5048 and the related malware-scan issues to understand the existing submitted step-result path and the interim pendingScanResult store. Trace how scan completion currently drives the Submitted → InReview gate. Done would require an agreed first-class ScanResult contract, configurable scanner aggregation, migration of the malware path, and policies for missing or unextractable results.