bug(compliance): make framework reports current, evidence-based, and non-certifying
Open
@parthrohit22 is already working on this.
Since Aug 22, 2026.
bug
core
priority: critical
- Dominant language
- Python
- Stars
- 57
- Forks
- 68
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 17
Description
What is wrong
Current reports can overstate assurance:
- A missing finding is counted as PASS, including failed/unrun rules.
- Every rule is forced into CIS, NIST, ISO and SOC 2 files even where the mapping is N/A, weak or inapplicable.
- Several SOC 2 descriptions are swapped or describe a different rule.
- The repository uses CIS Azure 2.0.0, NIST CSF 1.1 and ISO 27001:2013 metadata.
- Historical scans are reinterpreted using whatever mapping files happen to be deployed now.
- A technical cloud scanner cannot establish all organizational/process controls needed for certification.
Evidence: api/models/finding.py:535-595, compliance/frameworks/*.json, and .github/workflows/ci.yml:312-367.
Required work
Treat output as versioned technical evidence coverage, not certification. Build framework packs with explicit direct/supporting/unmapped rationale and independent review.
Acceptance criteria
- Depends on #263: PASS is emitted only from an explicit successful evaluation.
- Framework name, edition, mapping-pack version and source are persisted with every scan/report.
- Current supported framework editions are documented; legacy packs are clearly versioned.
- N/A and organizational controls are excluded from technical pass-rate denominators.
- Every mapping has rationale, evidence type, primary source, owner and review date.
- CI validates mapping semantics/fixtures, not only file presence.
- Public copy no longer claims full compliance or certification.
- A sample mapping set receives independent security/compliance review before release.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.