tlaplus / tlaplus/ValidationTestSuite
Expand VTS coverage for TLC soundness-critical qualification gaps
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Expand the current Validation Test Suite (VTS) so it can detect recently discovered TLC soundness bugs that may cause TLC to miss a real violation of a safety property (most severe), or report a bogus counterexample (less severe, but still qualification-relevant). These bugs are subtle and are not exposed by the current VTS strategy, which combines TLA+ language features pairwise. In practice, soundness bugs appear to require interactions among 3, 4, or more features. Exhaustively testing such higher-order combinations is generally infeasible due to combinatorial explosion.
This means the current VTS approach, while useful, is insufficient on its own for covering certain soundness-critical TLC behaviors relevant to safety-critical qualification (e.g., ISO 26262).
Critical soundness issues to drive coverage expansion:
- https://github.com/tlaplus/tlaplus/issues/1302
- https://github.com/tlaplus/tlaplus/issues/742
- https://github.com/tlaplus/tlaplus/issues/1145
- https://github.com/tlaplus/tlaplus/issues/1112
Required work
Characterize the bug-triggering interactions
- Identify which combinations of language features / semantic conditions are implicated in each issue.
- Document why pairwise composition is insufficient in each case.
Design targeted, non-exhaustive test strategies
- Develop test-generation strategies that go beyond pairwise coverage without requiring full n-wise enumeration.
- Candidate approaches may include:
- bug-class regression tests
- targeted higher-order feature combinations
- metamorphic tests
- constraint-guided test generation
- risk-based prioritization of combinations
Promote critical issues into permanent regression coverage
- Add minimized reproducer specs (where possible) to VTS.
- Ensure each reproducer is tied to a documented bug class and expected behavior.
Update qualification claims and coverage rationale
- Explicitly describe what kinds of soundness risks are now covered.
- Document residual risks from untested higher-order feature interactions.
Outcome needed: a VTS expansion plan that prioritizes detection of soundness-critical TLC failures beyond pairwise feature interaction testing, starting from the known issue set above.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the current VTS strategy and the linked TLC issues 1302, 742, 1145, and 1112 to characterize their triggering interactions. Done means a prioritized expansion plan, minimized permanent regression reproducers where possible, documented bug classes and expected behavior, and updated qualification coverage and residual-risk rationale.
Written by the indexing model from the issue text.
Assessment
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100