[Bug]: Compliance reports conflate unassessed controls with satisfied controls

Package

agent-mesh

Description

ComplianceEngine.generate_report() cannot distinguish a control that was actually assessed and satisfied from a control that was never assessed.

On current main, the Python compliance engine stores recorded violations, but does not persist which controls were successfully assessed.

Reporting derives controls_met from the complete framework control set minus controls with recorded violations: total = len(framework_controls), failed = len(violated_controls), met = total - failed, and score = (met / total * 100) when controls exist.

Because successful assessments are not persisted separately from controls that were never assessed, the reporting layer cannot distinguish:

• a control that was assessed and passed; from
• a control that was not assessed.

This matters because docs/specs/AUDIT-COMPLIANCE-1.0.md describes total_controls as the number of controls assessed and controls_met as controls fully satisfied.

The dashboard has the same semantic shape: DashboardAPI.get_compliance_report() derives controls_met from a framework-wide control count minus failed/denied audit entries.

I am not asserting that zero violations must mean zero compliance. The narrower issue is that the reporting model cannot distinguish zero violations after assessment from zero violations because the relevant controls were not assessed.

The Rust implementation also intentionally preserves 0 violations -> 100% in PR #2119, so the issue is not the score value by itself. The issue is the missing assessment-state distinction.

A resolution could either track assessment coverage explicitly, or redefine/rename the reporting fields so unassessed controls cannot be represented as "fully satisfied" or "evaluated".

How does this impact your work?

This degrades the reliability of compliance and assurance reporting.

A downstream consumer cannot tell whether a positive result is backed by actual control assessment or only by the absence of recorded violations.

The same distinction can propagate into dashboard reporting and Annex IV technical documentation as Controls evaluated / Controls met.

A consumer can work around this only by maintaining separate assessment-coverage state outside the current report model.

Timeline

No response

Steps to Reproduce

1. Check out AGT commit 5fe6bea36060de3ef35e9b5f3867bfd360a49d8c and install agent-mesh from that checkout.

2. Run the following without calling check_compliance() first:

from datetime import datetime, timedelta, timezone

from agentmesh.governance import ComplianceEngine, ComplianceFramework

engine = ComplianceEngine([ComplianceFramework.SOC2])

now = datetime.now(timezone.utc)

report = engine.generate_report(
    framework=ComplianceFramework.SOC2,
    period_start=now - timedelta(days=30),
    period_end=now,
)

print("total_controls:", report.total_controls)
print("controls_met:", report.controls_met)
print("controls_partial:", report.controls_partial)
print("controls_failed:", report.controls_failed)
print("compliance_score:", report.compliance_score)

No check_compliance() call has been made before generating the report.

With the current implementation, SOC 2 has two configured controls, so the reporting calculation is equivalent to:

total_controls: 2
controls_met: 2
controls_partial: 0
controls_failed: 0
compliance_score: 100.0

Actual behavior

Controls for which no assessment state exists can contribute to controls_met, because reporting only has recorded violations available.

Expected behavior

The reporting model should preserve the distinction between controls that were actually assessed and passed and controls that were not assessed, or the public field/spec terminology should explicitly describe the result as violation-based rather than assessment-based.

Environment

Repository: `microsoft/agent-governance-toolkit`

Source revision checked:
`5fe6bea36060de3ef35e9b5f3867bfd360a49d8c`

Affected package:
`agent-governance-python/agent-mesh`

Affected implementation:
`agentmesh.governance.compliance.ComplianceEngine`

Also checked:
- `agentmesh.dashboard.api.DashboardAPI`
- `agentmesh.governance.annex_iv.TechnicalDocumentationExporter`
- `docs/specs/AUDIT-COMPLIANCE-1.0.md`

### Logs / Error Output

```shell
No exception is raised. This is a reporting-semantics issue rather than a runtime error.

The problematic state is a positive compliance result when assessment coverage is not represented.

Code of Conduct

• I agree to follow the Microsoft Open Source Code of Conduct

Start with agentmesh.governance.compliance.ComplianceEngine.generate_report() and reproduce the unassessed SOC2 report using the Python snippet in the issue. Then compare agentmesh.dashboard.api.DashboardAPI.get_compliance_report(), TechnicalDocumentationExporter, and docs/specs/AUDIT-COMPLIANCE-1.0.md. Done means the reporting model or its terminology distinguishes assessed controls from unassessed controls across the affected outputs.