microsoft / microsoft/agent-governance-toolkit
[Bug]: promptDefenseGrade does not reflect operator configuration
- Dominant language
- Python
- Stars
- 6.3k
- Forks
- 1.1k
- Avg merge
- 5d 11h
- Merged PRs (30d)
- 142
Description
## Package
Other — `agent-governance-opencode`
## Description
`getPolicyStatus` reports `promptDefenseGrade: "A"`, `coverage: 12/12`, and no missing defenses across materially different operator configurations. `minimumPromptDefenseGrade` sits alongside this status in the policy schema, which makes the grade look like assurance about the supplied configuration even though it appears to describe a constant built-in detector catalogue.
Observed configurations all returned the same result:
| configuration | grade | coverage |
|---|---:|---:|
| no `additionalContext` key | A | 12/12 |
| `additionalContext: []` | A | 12/12 |
| `additionalContext: ["probe"]` | A | 12/12 |
| explicitly unsafe context instructions | A | 12/12 |
`promptDefenseBlocking` remained false throughout.
## How does this impact your work?
The session-start line prints `promptDefense=A`, and users may cite it as evidence that their configuration is hardened. We currently label it “advisory only; independent of configuration” and do not use it as a gate.
## Steps to Reproduce
1. Load policies with the four `additionalContext` variants above.
2. Call `getPolicyStatus` for each policy.
3. Compare `promptDefenseGrade`, coverage, missing defenses, and `promptDefenseBlocking`.
## Suggested resolution / acceptance criteria
Either:
- calculate a grade that reflects the operator-supplied configuration and enforce `minimumPromptDefenseGrade`; or
- rename the status field and document that it reports built-in catalogue coverage rather than configuration assurance.
The session-start log and status response should make the distinction unambiguous, with regression tests for materially weak and strong configurations.
## Environment
- `@microsoft/agent-governance-opencode@5.0.0`
- Node 26.7.0
- macOS arm64
- Verified against AGT commit `81955d4`
## Code of Conduct
- [x] I agree to follow the Microsoft Open Source Code of Conduct
## Coordination status
- Active implementation: #3677.
- Related follow-up: #3669 covers session-scoped state and staged evaluation; it is not implemented by #3677.
Please coordinate on #3677 rather than open a parallel fix for this issue.
Contributor guide
Research direction
Start with getPolicyStatus and the session-start log described in the issue, then review active implementation #3677 rather than opening a parallel fix. Done means the status and log clearly distinguish configuration assurance from built-in catalogue coverage, with regression tests for materially weak and strong configurations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js
- Domain
- ai, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100