OWASP / OWASP/openshield

bug(compliance): make framework reports current, evidence-based, and non-certifying

Open
#302 1 comment 0 reactions 1 assignee View on GitHub

@parthrohit22 is already working on this.

Since Aug 22, 2026.

bug core priority: critical
Dominant language
Python
Stars
57
Forks
68
Avg merge
3d 15h
Merged PRs (30d)
17

Description

What is wrong

Current reports can overstate assurance:

  • A missing finding is counted as PASS, including failed/unrun rules.
  • Every rule is forced into CIS, NIST, ISO and SOC 2 files even where the mapping is N/A, weak or inapplicable.
  • Several SOC 2 descriptions are swapped or describe a different rule.
  • The repository uses CIS Azure 2.0.0, NIST CSF 1.1 and ISO 27001:2013 metadata.
  • Historical scans are reinterpreted using whatever mapping files happen to be deployed now.
  • A technical cloud scanner cannot establish all organizational/process controls needed for certification.

Evidence: api/models/finding.py:535-595, compliance/frameworks/*.json, and .github/workflows/ci.yml:312-367.

Required work

Treat output as versioned technical evidence coverage, not certification. Build framework packs with explicit direct/supporting/unmapped rationale and independent review.

Acceptance criteria

  • Depends on #263: PASS is emitted only from an explicit successful evaluation.
  • Framework name, edition, mapping-pack version and source are persisted with every scan/report.
  • Current supported framework editions are documented; legacy packs are clearly versioned.
  • N/A and organizational controls are excluded from technical pass-rate denominators.
  • Every mapping has rationale, evidence type, primary source, owner and review date.
  • CI validates mapping semantics/fixtures, not only file presence.
  • Public copy no longer claims full compliance or certification.
  • A sample mapping set receives independent security/compliance review before release.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.