feat: native token-reduction output filter (PostToolUse result summarizer) — inspired by rtk

Summary

Proposal to implement a native, in-process output filter that compresses verbose tool results before they reach the model — reducing token consumption 60–90% on the noisiest commands (go test, golangci-lint, git status/diff, etc.) without changing the agent loop, widening any port, or introducing an external binary dependency.

Inspired by rtk-ai/rtk (Rust Token Killer), a CLI proxy that intercepts Bash commands and filters their output. After a deep-dive evaluation (architecture, security, feasibility — see the analysis below), the recommendation is not to adopt rtk as-shipped, but to re-implement the concept natively in Go as an operator-tier PostToolUse hook, reusing the exact mechanism the modelhook guardrail adapter already provides.

Why not adopt rtk directly

rtk is an external Rust binary that rewrites commands (git status → rtk git status) via agent hooks, then filters output. Five mecatl invariants collide with this design:

#	rtk mechanism	mecatl invariant violated	Severity
1	Post-authorization command rewriting (git status; rm -rf / — the gate never sees the rm)	The substitution-aware Bash gate (CWE-862 bypass)	High
2	External binary intercepting ALL Bash output	"No spawned exec interceptors" (the stdio-MCP rationale)	High
3	Project-local .rtk/filters.toml	Guardrails are operator-tier-only — a project filter stripping `WARNING	secret
4	Silent elision of output lines	The visible-redaction-marker discipline (guardrailRedactionMarker)	Medium
5	Mutated-producing hook sibling to the guardrail	mergeOutcomes ordering — a compression hook running after a security block could overwrite the block	High

rtk is also unpinned when installed via brew (auto-updates), outside govulncheck/SBOM coverage, and sits in a different language (Rust) — a supply-chain surface mecatl's discipline doesn't cover.

The proposed design — native, in-process, operator-tier

The seam already exists. PostToolUse hooks can rewrite tool results via HookOutcome.Mutated (engine/governance/hookevent.go). The modelhook guardrail adapter (issue #27) already does this for security. A token-reduction filter is the same mechanism for a different purpose (compression, not blocking).

Architecture

Agent runs `go test ./...` (gate authorizes normally)
  → Bash tool executes, captures stdout (e.g. 50KB of test output)
  → PostToolUse hook fires
  → output filter (wired as the INNER of the modelhook decorator)
     applies: error-only + dedup + go-test-NDJSON parse
  → result recorded as ~2KB compressed summary with [filtered: N lines elided] marker
  → model sees the compressed result on this turn AND every replay turn

Six constraints (from the security analysis)

1. No command rewriting. Compression applies to results only, never to the command string. The permission gate classifies the exact string the shell executes. (Closes the CWE-862 bypass.)
2. Filtering is a PostToolUse hook wired as the inner of the modelhook decorator, so mergeOutcomes guarantees a security block overwrites any compression. Never a sibling hook outside the guardrail chain.
3. Filters are operator-tier only — read from user-global settings.yaml, strictly parsed, project-local filter files ignored with a WARN (mirrors permconfig.Resolver.OperatorGuardrails()).
4. Every filtered result carries a visible [filtered: …] marker (reuse the guardrailRedactionMarker discipline) and is bounded by maxContentBytes-equivalent limits with fail-open/closed semantics.
5. No external binary. In-process Go, stdlib-only leaf (like envscrub), covered by govulncheck and the depguard allowlist.
6. Stats-only modes are forbidden for decision-driving commands (go test, git diff, git status); dedup-with-counts is the most aggressive permitted transform on those.

Filtering strategies (ported from rtk's taxonomy, in Go stdlib)

Strategy	Implementation	Reduction	Commands
Error-only	Keep stderr, drop stdout	60–80%	test runners, builds
Deduplication	Collapse repeated lines with counts [ERROR] … (×5)	70–85%	logs, repeated test failures
Grouping	Group errors by rule/file → counts	80–90%	golangci-lint, grep
Stats extraction	Count/aggregate, drop details	90–99%	git status, git log
Declarative line filters	strip/keep regex, max_lines, tail_lines, truncate_lines_at (YAML, not TOML)	60–80%	long-tail commands

MVP scope (~2 engineer-weeks)

3 dedicated parsers (the highest-value commands for a Go harness):

1. go test -json — NDJSON line walk, keep FAIL/--- FAIL/panic + package summary
2. golangci-lint --out-format json — group by linter field → counts
3. git status / git diff --stat — porcelain parsing → "3 modified, 1 staged"

4 generic strategies (command-agnostic, never rot):

• Error-only, dedup, declarative strip/keep/lines (YAML), structure-only (JSON schema extract)

Config surface

A filters: YAML subtree in operator-tier settings.yaml, parsed via the existing strict-YAML path in permconfig — not TOML (mecatl is a YAML house; go-toml is only a transitive dep via toolhive and mecatl never imports it). Example:

filters:
  enabled: true
  # Generic declarative filters (regex-based, command-agnostic)
  rules:
    - match: "^go test"
      strategy: go-test-json
    - match: "^golangci-lint"
      strategy: golangci-json
    - match: "^git status"
      strategy: git-status-stats
    - match: "^make\\b"
      strip_lines: ["^make\\[", "^\\s*$"]
      max_lines: 40

Why YAML not TOML

• mecatl's house config format is YAML (settings.yaml, permconfig, strict unknown-key parsing)
• go-toml is only a transitive dep (via toolhive); mecatl never imports it — promoting it to direct for one feature is a gratuitous dep
• The engine module boundary: engine/go.mod has a deliberately tiny closure; YAML parsing for filters belongs in internal/adapter (composition), keeping the engine clean

Where the code lives

• internal/adapter/outfilter/ — the filter adapter (implements port.HookRunner), wired into the main engine's PostToolUse chain as the inner of the modelhook decorator (composition, internal/app/build.go)
• Parsers are Go stdlib (encoding/json, regexp, bufio, strings) — no new direct deps
• Config parsed via the existing permconfig strict-YAML path

Token savings estimate (from rtk's benchmarks, adjusted for a Go harness)

Command	Frequency/30min session	Standard tokens	Filtered tokens	Savings
go test	3x	6,000	600	-90%
golangci-lint	2x	4,000	600	-85%
git status	10x	3,000	600	-80%
git diff	5x	10,000	2,500	-75%
grep/rg	8x	16,000	3,200	-80%
Total		~39,000	~7,500	-81%

And critically — unlike rtk which only saves on the current turn — mecatl replays tool results on every subsequent turn (RecordToolResults → conversation history). So a compressed result saves tokens on EVERY future turn in the session, compounding the savings.

What this does NOT do

• ❌ Rewrite commands (no git status → rtk git status) — the gate sees the exact string
• ❌ Introduce an external binary — in-process Go only
• ❌ Honor project-local filter config — operator-tier only
• ❌ Silently elide — every filter emits a visible [filtered: …] marker
• ❌ Run outside the guardrail chain — wired as the inner of the modelhook decorator so security always wins
• ❌ Widen port.LLMRequest or any port — uses the existing HookOutcome.Mutated mechanism
• ❌ Touch engine/ — lives entirely in internal/adapter/ + composition

Acceptance criteria

• A PostToolUse output-filter adapter exists in internal/adapter/outfilter/, implementing port.HookRunner
• Wired as the inner of the modelhook decorator in composition, so mergeOutcomes guarantees security-wins on conflict
• 3 dedicated parsers: go test -json, golangci-lint --out-format json, git status/git diff --stat
• 4 generic strategies: error-only, dedup, declarative strip/keep/lines (YAML), structure-only
• Config in operator-tier settings.yaml under filters:, strict-parse, project-local ignored with WARN
• Every filtered result carries a [filtered: …] marker
• Bounded by maxContentBytes-equivalent with fail-open/closed
• No new direct Go dependencies (stdlib only)
• Offline tests with fixtures for each parser
• task lint && task test green; go run ./cmd/mecademo unchanged (filter is opt-in, off by default)

Downstream / future

• The generic declarative filters (strip/keep/max_lines) are community-contributable via YAML without code changes
• If the parser set grows past ~10, that's the trip-wire to extract a shared abstraction (the same discipline as the delegation-observability ChildActivity trip-wire)
• The [filtered: …] marker could surface as a diagnostic (like the guardrail "checker DOWN" WARN) if a filter ever degrades

References

• rtk-ai/rtk — the inspiration (Apache-2.0, Rust)
• internal/adapter/modelhook/ — the existing guardrail adapter whose HookOutcome.Mutated + mergeOutcomes pattern this follows
• docs/adr/0021-guardrails.md — the guardrails ADR (operator-tier-only config discipline this mirrors)
• Issue #27 — guardrails (the modelhook adapter precedent)