microsoft / microsoft/agent-governance-toolkit
contributor_check and credential_audit have diverged into two copies; CI runs the one without retry, the packaged one lacks all hardening
- Dominant language
- Python
- Stars
- 6.3k
- Forks
- 1.1k
- Avg merge
- 5d 11h
- Merged PRs (30d)
- 142
Description
`contributor_check.py` and `credential_audit.py` exist in two copies that have
diverged in **both** directions for three months. Each copy has real
functionality the other is missing, and the copy that runs in CI is not the
copy that would ship to pip users.
## How it happened
| Date | Change |
|---|---|
| 2026-05-03 | #1711 copied `scripts/contributor_check.py` and `scripts/credential_audit.py` into `agent_compliance/cli/` to expose them as pip console scripts |
| 2026-05-12 | #2196 added exponential-backoff retry on `URLError`/5xx to the **packaged** `credential_audit`. Never back-ported to `scripts/` |
| 2026-06-12 | #2852 added org-aware and domain-aware hardening to **`scripts/`** `contributor_check`. Never forward-ported to the package |
No sync mechanism, no test asserting the two agree, so neither omission was
noticed.
## Current divergence
`contributor_check.py`: `scripts/` is 1475 lines, packaged is 1076. Both define
the same ten signal functions, so it looks equivalent at a glance. It is not.
**All eleven hardening functions are absent from the packaged copy:**
```
_org_owned_established _has_prior_target_contribution
_count_maintainer_merged _is_public_org_member
_load_allowlist _is_allowlisted
_apply_allowlist _dampen_for_established_accounts
_established_credibility _is_established_repo_aged
_user_contributed_to
```
That is the entire established-account dampening path, the org-backed and
prior-interaction credibility tiers, the maintainer allowlist, and the
star-buying and merge-ring guards.
`credential_audit.py`: `scripts/` is 392 lines, packaged is 456. The packaged
copy has `_retry_sleep_seconds` and retry/backoff that `scripts/` does not,
covered by `tests/test_cli_api_retry.py`.
Also packaged-only or scripts-only, not merely drifted:
- `cluster_detect.py` exists in `scripts/` only. No packaged equivalent.
- `contributor_check_action.py`, the orchestrator that posts the comment and
applies labels, exists in `scripts/` only.
## Why this matters now
**1. Production is running the copy without retry.**
`.github/actions/contributor-check` resolves its scripts through
`$ACTION_PATH/../../../scripts/`, so every contributor check in CI runs the
`scripts/` copy. That copy lacks the #2196 backoff.
This got worse recently. #2950 made `UNKNOWN` fail-closed and moved it above
`LOW`/`MEDIUM` in `RISK_ORDER`, so a check that errors now surfaces a comment
and a `needs-review:UNKNOWN` label instead of being silently dropped. The check
routinely brushes GitHub rate limits: it makes tens of calls per run, several
against the Search API, which is capped at 30 requests per minute. Correct
behaviour, except the code path that would absorb a transient 403 or 5xx is
sitting in the other copy. The likely result is spurious `UNKNOWN` flags on
legitimate contributors.
**2. The package would be a regression if published.**
`agent-governance-toolkit-compliance` declares the console scripts:
```
agt-contributor-check = agent_compliance.cli.contributor_check:_entry
agt-credential-audit = agent_compliance.cli.credential_audit:_entry
```
It is at `5.0.0` in-repo and returns 404 on PyPI, so it has never actually been
released. If it were released today, `pip install` users would get contributor
screening with no dampening for established accounts and no allowlist escape
hatch: more false positives on legitimate contributors, and no way to soften a
misfire. Tutorial 53 documents the dampening and allowlist behaviour, so the
docs would describe the `scripts/` copy while the package behaved differently.
**3. It blocks moving consumers off cross-repo checkout.**
Downstream consumers currently `actions/checkout` this repo at a pinned SHA to
get `scripts/`, inside `pull_request_target` workflows holding write tokens. A
published package would let them `pip install` a pinned version and drop the
checkout entirely, which is a real supply-chain improvement. That is not
available while the packaged copy is behind.
## Suggested fix
One source of truth. `agent_compliance.cli` is the natural home since it
already has the packaging, entry points, and tests:
1. Port the eleven hardening functions and the `#2852` behaviour into
`agent_compliance/cli/contributor_check.py`.
2. Keep the packaged `credential_audit` retry path, since it is strictly ahead.
3. Reduce `scripts/contributor_check.py` and `scripts/credential_audit.py` to
thin shims importing from `agent_compliance.cli`, preserving their current
CLI surface so the composite action and Tutorial 53 keep working unchanged.
4. Decide on `cluster_detect.py` and `contributor_check_action.py`: either
package them too, or state deliberately that they stay repo-local tooling.
5. Add a test that fails if the two entry points disagree, so this cannot
silently recur.
Step 3 is the part worth debating. The alternative is to delete the packaged
copy and package `scripts/` instead, which is less churn now but throws away
the existing tests and entry points.
Happy to do the work if the approach is agreed. Raising it as an issue first
rather than opening a PR, since consolidating a published package's public
surface seems like a maintainer call.
Found while writing this control up as an RFC for the CoSAI WS4 workstream
(`cosai-oasis/ws4-secure-design-agentic-systems#156`).
Contributor guide
Research direction
Start by comparing scripts/contributor_check.py and agent_compliance/cli/contributor_check.py, then review the corresponding credential_audit.py files and tests/test_cli_api_retry.py. Check the console-script entry points and .github/actions/contributor-check path before deciding how cluster_detect.py and contributor_check_action.py are handled. Done means one maintained implementation, preserved CLI behavior, aligned retry and hardening behavior, and a regression test preventing future drift.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, cli, security
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100