microsoft / microsoft/agent-governance-toolkit

normalize.rs: nested encodings decode in only one order (Rust twin of #3502)

Open
#3,521 2 comments 0 reactions 0 assignees View on GitHub
needs-review:HIGH
Dominant language
Python
Stars
6.3k
Forks
1.1k
Avg merge
5d 11h
Merged PRs (30d)
142

Description

### Summary

`agentmesh::normalize::try_decode_once` has the same nesting-order gap that #3502 reports for the Python `agent_os.normalize`: an ambiguous outer layer (percent / unicode-escape / HTML entity) wrapping a base64 or hex blob is left verbatim, with no transform tag, while the reverse nesting unwraps fully.

This is filed as the Rust half of #3502 so the cross-SDK parity the module promises is tracked rather than assumed. PR #3503 fixes the Python side only.

### Where

`agent-governance-rust/agentmesh/src/normalize.rs`, the acceptance condition on all three ambiguous layers:

- line 428 — percent
- line 440 — unicode escape
- line 451 — HTML entity

Each reads:

```rust
&& english_score(&dec) > english_score(trimmed)
```

### Why it lets nesting survive

The three layers are ambiguous: their markers occur in benign text (`50% off`, `&`, a Windows path), so a decode has to justify itself, and an English-marker gain is the justification when the payload is prose.

It is not the only one. When the revealed text is *itself* an encoded blob, both sides score zero English markers, so a gain can never be shown and the outer layer never fires. The base64/hex layer immediately below accepts on the printable ratio alone — the comment there says this is "so nested encodings unwrap" — so an outer layer that reveals a blob that layer would take is the same kind of progress, and currently gets the opposite treatment.

The consequence is not only that the payload stays encoded. No transform fires either, so a caller auditing `transforms` sees a clean pass rather than a rejected decode: the incompleteness is invisible to policy.

### Expected

`percent(base64(payload))` and `base64(percent(payload))` should both unwrap to the same canonical text, and both should report both transforms. Same for hex, and for the escape and entity layers.

### Suggested fix

Mirror what PR #3503 does for Python: alongside the English-gain test, accept when the decode revealed a blob the base64/hex layer would actually attempt. Two details from that PR that are worth carrying over:

1. The predicate must use the base64/hex layer's *real* preconditions, including `len % 4 == 0` for base64 and `len % 2 == 0` for hex. An alphabet-and-length-only check counts a 19-char base64-alphabet run as a benefit even though no decoder would attempt it, which loosens the guard for nothing.
2. This cannot loosen the guard into benign prose: a contiguous, non-whitespace, >= 16 char run of one alphabet is strictly narrower than arbitrary printable text.

The Python change is `_has_decode_benefit` / `_decode_attemptable` in `agent-governance-python/agent-os/src/agent_os/normalize.py`.

### Note on verification

The line numbers and the reasoning above come from reading `normalize.rs` against the Python module it mirrors — the two are structurally identical at these three sites. I did not run the Rust suite (no toolchain in the environment I checked from), so the runtime behaviour is inferred from the shared structure rather than reproduced. Worth confirming with a test before fixing.

The Python test that pins the behaviour is `TestNestingOrderSymmetry` in `agent-governance-python/agent-os/tests/test_normalize.py`; the Rust suite in `normalize.rs` mirrors that suite case-by-case, so the parity cases can be ported directly.

Contributor guide

Open the contributing guide

Research direction

Start in agent-governance-rust/agentmesh/src/normalize.rs at lines 428, 440, and 451, then compare the corresponding _has_decode_benefit and _decode_attemptable logic in agent-governance-python/agent-os/src/agent_os/normalize.py. Port the nesting-order parity cases from TestNestingOrderSymmetry in agent-governance-python/agent-os/tests/test_normalize.py to the mirrored Rust suite, and run the Rust tests. Done means both nesting orders unwrap to the same canonical text and report both transforms for percent, hex, escape, and entity layers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, rust
Domain
security
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.