finos / finos/common-cloud-controls

Proposal: CCC.Monitor control for validating ML-based detection before reliance

Open
#1,214 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
87
Forks
80
Avg merge
4d 15h
Merged PRs (30d)
13

Description

CCC.Monitor controls cover the presence and protection of monitoring. None covers whether a detection mechanism detects. Two failure modes motivate this.

Operating-point metrics move with class prevalence, so scores from vendor benchmarks do not transfer to production rates. And detectors can score acceptably while discriminating worse than chance, measured at 3 of 8 widely used methods in a controlled study (https://doi.org/10.1109/ACCESS.2026.3705430).

Proposed control, drafted to CCC conventions on request.

- ML-based detection used to satisfy a monitoring control MUST be validated against labeled data with threshold-independent metrics before reliance.
- Reported performance MUST state the anomaly prevalence of the evaluation data and MUST be compared against the predict-all baseline at that prevalence.
- Validation MUST be repeated when the telemetry distribution changes materially.

Mapped threat, missed detection due to unvalidated detection tooling. Happy to draft the YAML and the threat mapping if the working group wants this.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing CCC.Monitor controls and their conventions, then examine how threats are mapped. No files or tests are named; done would mean agreement on the proposed validation control, its YAML representation, and the mapped threat, with the working group's decision reflected.

Written by the indexing model from the issue text.

Assessment

Tech stack
yaml
Domain
security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.