microsoft / microsoft/agent-governance-toolkit

Example manifests: 32 of 42 still leave adapter-evaluated intervention points unbound (post-#3528)

Open
#3,540 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.3k
Forks
1.1k
Avg merge
5d 11h
Merged PRs (30d)
142

Description

Follow-ups from the #3528 review (the PR fixes the migrator and 8 example manifests; these are the parts it deliberately leaves for later).

**1. 32 of 42 example manifests still leave `pre_tool_call` (and other evaluated points) unbound**

Verified by parsing every manifest's `intervention_points` at #3528 head `5a75c2bf`:

- `examples/policies/african-regulatory/*` (15 manifests) — bind `input`/`output` only
- `examples/policies/india-regulatory/*` (5) — same
- `examples/policies/production/*` (5) — same
- `examples/policies/lotl_prevention_policy.yaml` — `input` only
- `examples/policy-templates/*` (5 input/output-only; `wire-protocol-rules.yaml` binds `pre_tool_call` only)

16 of the 19 in-repo adapters evaluate `pre_tool_call` (all except a2a, guardrails, llamaindex), and the engine deliberately denies unbound points with `runtime_error:intervention_point_unknown` — so an agent loading any of these manifests through those adapters fails on its first tool call. This is the same defect class #3528 fixes in the crewai/maf/openai-agents/smolagents examples; #3528's known-gaps note attributes it to the `agt policy gen` output target, but the missing bindings need fixing regardless of which tool emitted them.

Suggested fix: bind every point the adapters evaluate in each of these manifests (deny/allow semantics matching each pack's intent — regulatory packs likely want their pattern rules applied at `pre_tool_call`/`post_tool_call` too, which is also the safer direction), and add a CI lint that walks `examples/**/manifest*.yaml` + `examples/policy-templates/**` and fails when a manifest omits a point the shipped adapters evaluate.

**2. No test pins the `notes` advisory channel**

The second vacuous-success fix in #3528 (confidence-threshold advisory emitted once, surviving `--write` without duplication, rendered in the report) was verified empirically during review but has no regression test in `agent-governance-python/agt-policies/tests/test_migrate.py`. A test should pin: advisory present exactly once after a plain run, after `--write`, and in the rendered report.

**3. `BREAKING_CHANGES.md` wording (minor)**

The "a manifest left on `$.tool_result` … stops denying" sentence overstates breakage: a migrated leaf-walking policy retargeted to `$.tool_result` still denies pattern matches; only policies matching the raw target as a single string stop matching. Safe direction, but worth precise wording.

Found while reviewing #3528; all verified against head `5a75c2bf` with live OPA probes.

Contributor guide

Open the contributing guide

Research direction

Start with the manifests under examples/policies and examples/policy-templates, then inspect agent-governance-python/agt-policies/tests/test_migrate.py and the CI configuration. Verify which intervention points the shipped adapters evaluate, add coverage for the notes advisory channel, and review BREAKING_CHANGES.md. Done means the affected manifests bind the evaluated points, CI catches omissions, the advisory is pinned through writing and rendering, and the wording is precise.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.