feature: Robustness suite — semantic tier with human sign-off
@Amitfre15 is already working on this.
Since Aug 26, 2026.
- Dominant language
- Python
- Stars
- 302
- Forks
- 107
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 41
Description
Parent
#2087
What to build
Extend the Robustness suite with the semantic tier per docs/specs/eval-framework.md §4: full paraphrase/rewording perturbations, for both the invariance and sensitivity families. Semantic perturbations may be LLM-drafted, but every one requires human sign-off confirming it actually preserves (invariance) or changes as intended (sensitivity) the meaning before it enters the corpus — the human review is the real ground-truth authority; the LLM only saves drafting time.
The sensitivity family's edit-type coverage carries over from the mechanical tier (#2466) — including the restriction-word case ("only", "just" narrowing or broadening a grant) — but expressed as natural paraphrase instead of literal word insertion (e.g. rewording a grant to "solely" or "exclusively" cover a subset, rather than mechanically inserting "only").
Acceptance criteria
- Semantic-tier invariance and sensitivity perturbations exist for the primary scenario set, drafted with LLM assistance.
- The semantic-tier sensitivity set includes at least one paraphrased restriction case (e.g. "solely"/"exclusively"/equivalent narrowing language), not just the mechanical-tier's literal "only"/"just" insertion.
- A documented sign-off step/record exists and is required before a semantic perturbation is added to the corpus — no semantic perturbation ships without it.
- The suite runs both tiers (mechanical + semantic) and both families, still scored as the same two separate metrics.
Blocked by
#2466 (needs the mechanical-tier harness and the invariance/sensitivity scoring it introduces).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.