thefrontside / thefrontside/effectionx
Test policy validation against historical pull requests
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 12
- Forks
- 4
- Avg merge
- 3h 17m
- Merged PRs (30d)
- 1
Description
Parent Quest: #254
Story
As an effectionx maintainer, I want to replay policy validation against known historical defects and their fixes, so we know the complete XMD mechanism distinguishes real violations from compliant code.
Current gap
Individual rules and prompts can be tested with synthetic examples, but those examples do not show that policy discovery, change selection, validator invocation, and verdict reporting work together on changes that actually occurred in effectionx.
PRs #226 and #252 contain documented before-and-after cases for two existing policies. Their revisions are currently external history rather than stable verification inputs owned by the policy suite.
Contract
Create a repository-owned historical corpus that exercises policy validation through the same XMD entrypoints used by contributors.
The initial corpus pins these revision pairs:
| Policy | Pull request | Violating revision | Fixed revision |
|---|---|---|---|
| Async Teardown | #226 | 02d93f965a0b43d672de4f1dc577993dc260d537 |
2367e02df5f24453328ed7d2e41f46d960ef40a9 |
| Scope-Bound Event Registration | #252 | b1139b4f65b1362ceb5a71bd672c69fb51d8d023 |
aae99a4c30cd89092142b9708183378ef49becee |
Fixtures preserve only the repository inputs needed to reproduce each verdict and record their source revisions. Automated verification does not fetch mutable pull-request state or depend on GitHub availability.
Each historical scenario runs the applicable policy’s public XMD Validate entrypoint. It records the selected policy, validator, expected verdict, actual verdict, and evidence. The corpus also contains a clean negative control that would fail if a validator reported every inspected change.
Acceptance
- The violating revision from PR #226 fails the Async Teardown policy, and its fixed revision passes.
- The violating revision from PR #252 fails the Scope-Bound Event Registration policy, and its fixed revision passes.
- Each result is produced through the policy’s XMD entrypoint rather than a private rule-test shortcut.
- A clean negative control passes and produces no policy violation.
- Swapping an expected failing fixture for its fixed counterpart causes the corresponding regression assertion to fail, demonstrating that the corpus tests the verdict rather than command completion alone.
- Historical verification runs without network access and produces the same selected validators, findings, and exit statuses for unchanged inputs.
- Fixture provenance identifies the exact pull request and revision without requiring readers to reconstruct the source manually.
Evidence
A single repository command runs the historical corpus and reports every scenario by policy and source revision. CI executes that command before the README policy workflow can be considered delivered.
Dependencies
- Depends on #257 for public per-policy XMD entrypoints.
- Uses lint rules delivered by #256.
- Blocks README and CI policy composition in #254.
Out of scope
- Replaying every historical pull request
- Fetching GitHub data during tests
- Benchmarking AI models
- Deno lint compatibility
- Treating synthetic rule fixtures as substitutes for end-to-end historical cases
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the public per-policy XMD Validate entrypoints from #257 and the lint rules from #256, then trace how a repository-owned historical corpus can run without network access. Add fixtures for the four pinned revisions, the clean negative control, and their provenance. Done means the repository command reports matching validators, findings, verdicts, and exit statuses, and swapping a failing fixture for its fixed counterpart fails regression verification.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ci-cd, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100