microsoft / microsoft/agent-governance-toolkit

[Bug]: lint-policy accepts a rule whose condition can never match, and the evaluator silently applies the default

Open
#3,661 5 comments 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Python
Stars
6.3k
Forks
1.1k
Avg merge
5d 11h
Merged PRs (30d)
142

Description

### Package

agent-os

### Description

`agt lint-policy` is described as linting policy files "for common mistakes", and `docs/LIMITATIONS.md` section 9 covers governance that is not enabled — no policies loaded, or `permissive` mode. There is a case in between that neither catches: governance **is** enabled, the policy **is** loaded, the rule **is** present and syntactically valid, and it still matches nothing that can ever occur.

Reproduction, on `agent-governance-toolkit` and `agent-os-kernel` from PyPI, Python 3.14, clean venv:

```python
from agent_os.policies import (
PolicyEvaluator, PolicyDocument, PolicyRule,
PolicyCondition, PolicyAction, PolicyOperator, PolicyDefaults,
)

def run(label, field, value):
ev = PolicyEvaluator(policies=[PolicyDocument(
name="t", version="1.0",
defaults=PolicyDefaults(action=PolicyAction.ALLOW),
rules=[PolicyRule(
name="block", action=PolicyAction.DENY, priority=100,
condition=PolicyCondition(field=field, operator=PolicyOperator.IN, value=value),
)],
)])
print(label, ev.evaluate({"tool_name": "delete_file"}).allowed)

run("tool_name (correct):", "tool_name", ["delete_file"])
run("toolname (typo): ", "toolname", ["delete_file"])
run("empty value set: ", "tool_name", [])
```

```
tool_name (correct): False
toolname (typo): True
empty value set: True
```

Expected: a `deny` rule either fires, or something reports that it cannot. Actual: cases two and three return `allowed=True` with `matched_rule=None` and `reason="No rules matched; default action applied"`. No exception and no warning. The YAML equivalents — a rule with a misspelled `field`, an empty `IN` set, and a `deny` rule made unreachable by a higher-priority blanket `allow` — all pass `agt lint-policy` with `No issues found`.

The reason this may be worth listing separately from section 9 is what it looks like to a reviewer. A missing policy file is visible: `agt doctor` and the dashboard can report it. A rule that reads correctly and covers nothing is visible nowhere — the file is present, the linter is green, the audit log shows ordinary allowed actions, and application tests pass because the tool itself works. The first honest signal is a denial that never happened and nobody was waiting for.

Severity depends on the default. Under `default_action: deny` this fails closed and the cost is a confusing refusal. Under `default_action: allow` it fails open, and that is the default in the quickstart policy in `README.md`, which is named `production-policy`.

Two static checks would cover most of it in `lint-policy`: warn when a condition's `field` is not one the evaluator can populate, so `toolname` is reported next to `tool_name`; and warn when a condition cannot be satisfied by construction — an empty `IN` set, or a rule shadowed by a higher-priority blanket `allow`.

Separately and much smaller: the quickstart policy in `README.md` has no `version` key, and `agt lint-policy` rejects it with `error: Missing required field 'version'`.

Happy to open a PR for either check if that is a direction you want.

### How does this impact your work?

Not blocking — there is an obvious workaround, which is to write the field names correctly. The reason I am reporting it rather than fixing my own typo is that this class of defect is invisible to review: the control is present, readable, approved, and enforces nothing, and every signal available to the operator says the policy is in force. That is the failure mode a governance layer exists to prevent, so I would expect the linter to be the place it gets caught.

### Timeline

_No response_

### Steps to Reproduce

1. python3 -m venv /tmp/agt
2. /tmp/agt/bin/pip install agent-governance-toolkit agent-os-kernel
3. Run the script in the description with /tmp/agt/bin/python
4. For the linter half: write the same three rules as YAML policy files with `apiVersion: governance.toolkit/v1`, `version: "1.0"`, `default_action: allow`, and run `agt lint-policy` against the directory

### Environment

- OS: macOS
- Python: 3.14.3 (fresh venv, no other packages)
- agent-os-kernel: 3.7.0
- agent-governance-toolkit: 4.1.0
- Installed from PyPI, default index

### Logs / Error Output

```shell
tool_name (correct): False
toolname (typo): True
empty value set: True

$ agt lint-policy ./policies
No issues found.
```

### Code of Conduct

- [x] I agree to follow the Microsoft Open Source Code of Conduct

Contributor guide

Open the contributing guide

Research direction

Start with the `agt lint-policy` entry point and the `agent_os.policies` evaluator classes shown in the reproduction; run the Python script and the YAML linting steps to confirm both behaviors. Trace how condition fields, empty `IN` values, and rule priority are represented, then add diagnostics for conditions that cannot match. Done means these cases are reported by the linter, and the README quickstart's missing `version` issue is addressed or explicitly separated.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.