feat: pilot Azure Policy rule-synthesis against AZ-STOR-002 as parity benchmark
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 57
- Forks
- 68
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 17
Description
What problem does this solve?
Before building any Azure-Policy-to-OpenShield-rule generation pipeline, we need a measured answer
to whether it actually saves reviewer time and produces semantically correct rules — not an
assumption. No benchmark exists today.
Describe the solution
- Pick one supported, direct-property Azure Policy class (not one requiring related-resource
traversal or complex aliasing). - Generate a draft rule bundle from that policy definition and compare it against the existing,
human-reviewed scanner/rules/az_stor_002.py as the golden reference — do not merge a duplicate
rule; this is a quality measurement exercise only. - Measure reviewer time to bring the generated bundle to mergeable quality, and count semantic
defects found in review. - No auto-merge under any circumstance — generated output always goes through the normal PR/review
path with a human owner.
Alternatives considered
Build the full synthesis pipeline first and evaluate later — rejected; every rule still needs
correct semantics, fixtures, playbook, and framework mappings regardless of how it was drafted.
Additional context
Closes the loop on the AZ-STOR-006 duplicate-rule situation from PR #242 and issue #245 —
reinforces "don't create a duplicate rule."
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scanner/rules/az_stor_002.py as the golden reference, then review issue #245 and PR #242 for the AZ-STOR-006 duplicate-rule context. Select one direct-property Azure Policy definition, produce a draft bundle for comparison, and record reviewer time and semantic defects; done means a measured result without merging a duplicate or enabling auto-merge.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- cloud, security, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100