finos / finos/common-cloud-controls
Proposal: CCC.Monitor control for validating ML-based detection before reliance
- Dominant language
- Go
- Stars
- 87
- Forks
- 80
- Avg merge
- 4d 15h
- Merged PRs (30d)
- 13
Description
CCC.Monitor controls cover the presence and protection of monitoring. None covers whether a detection mechanism detects. Two failure modes motivate this.
Operating-point metrics move with class prevalence, so scores from vendor benchmarks do not transfer to production rates. And detectors can score acceptably while discriminating worse than chance, measured at 3 of 8 widely used methods in a controlled study (https://doi.org/10.1109/ACCESS.2026.3705430).
Proposed control, drafted to CCC conventions on request.
- ML-based detection used to satisfy a monitoring control MUST be validated against labeled data with threshold-independent metrics before reliance.
- Reported performance MUST state the anomaly prevalence of the evaluation data and MUST be compared against the predict-all baseline at that prevalence.
- Validation MUST be repeated when the telemetry distribution changes materially.
Mapped threat, missed detection due to unvalidated detection tooling. Happy to draft the YAML and the threat mapping if the working group wants this.
Contributor guide
Research direction
Start by reviewing the existing CCC.Monitor controls and their conventions, then examine how threats are mapped. No files or tests are named; done would mean agreement on the proposed validation control, its YAML representation, and the mapped threat, with the working group's decision reflected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- yaml
- Domain
- security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100