microsoft / microsoft/agent-governance-toolkit
Example manifests: 32 of 42 still leave adapter-evaluated intervention points unbound (post-#3528)
- Dominant language
- Python
- Stars
- 6.3k
- Forks
- 1.1k
- Avg merge
- 5d 11h
- Merged PRs (30d)
- 142
Description
Follow-ups from the #3528 review (the PR fixes the migrator and 8 example manifests; these are the parts it deliberately leaves for later).
**1. 32 of 42 example manifests still leave `pre_tool_call` (and other evaluated points) unbound**
Verified by parsing every manifest's `intervention_points` at #3528 head `5a75c2bf`:
- `examples/policies/african-regulatory/*` (15 manifests) — bind `input`/`output` only
- `examples/policies/india-regulatory/*` (5) — same
- `examples/policies/production/*` (5) — same
- `examples/policies/lotl_prevention_policy.yaml` — `input` only
- `examples/policy-templates/*` (5 input/output-only; `wire-protocol-rules.yaml` binds `pre_tool_call` only)
16 of the 19 in-repo adapters evaluate `pre_tool_call` (all except a2a, guardrails, llamaindex), and the engine deliberately denies unbound points with `runtime_error:intervention_point_unknown` — so an agent loading any of these manifests through those adapters fails on its first tool call. This is the same defect class #3528 fixes in the crewai/maf/openai-agents/smolagents examples; #3528's known-gaps note attributes it to the `agt policy gen` output target, but the missing bindings need fixing regardless of which tool emitted them.
Suggested fix: bind every point the adapters evaluate in each of these manifests (deny/allow semantics matching each pack's intent — regulatory packs likely want their pattern rules applied at `pre_tool_call`/`post_tool_call` too, which is also the safer direction), and add a CI lint that walks `examples/**/manifest*.yaml` + `examples/policy-templates/**` and fails when a manifest omits a point the shipped adapters evaluate.
**2. No test pins the `notes` advisory channel**
The second vacuous-success fix in #3528 (confidence-threshold advisory emitted once, surviving `--write` without duplication, rendered in the report) was verified empirically during review but has no regression test in `agent-governance-python/agt-policies/tests/test_migrate.py`. A test should pin: advisory present exactly once after a plain run, after `--write`, and in the rendered report.
**3. `BREAKING_CHANGES.md` wording (minor)**
The "a manifest left on `$.tool_result` … stops denying" sentence overstates breakage: a migrated leaf-walking policy retargeted to `$.tool_result` still denies pattern matches; only policies matching the raw target as a single string stop matching. Safe direction, but worth precise wording.
Found while reviewing #3528; all verified against head `5a75c2bf` with live OPA probes.
Contributor guide
Research direction
Start with the manifests under examples/policies and examples/policy-templates, then inspect agent-governance-python/agt-policies/tests/test_migrate.py and the CI configuration. Verify which intervention points the shipped adapters evaluate, add coverage for the notes advisory channel, and review BREAKING_CHANGES.md. Done means the affected manifests bind the evaluated points, CI catches omissions, the advisory is pinned through writing and rendering, and the wording is precise.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100