tlaplus / tlaplus/ValidationTestSuite

Expand VTS scope to TLC liveness checking

Open
#9 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
6
Forks
2
PR merge metrics
No merged PRs in 30d

Description

Extend the VTS to include liveness-checking in addition to safety. This requires foundational/conceptual work to define an appropriate validation strategy for TLC’s liveness checking. Unlike safety validation, liveness cannot currently be tested by comparing TLC and Apalache outputs, because Apalache does not support liveness checking.

Potential approaches to evaluate:

  1. Reduce liveness to safety
  • Investigate whether a liveness-to-safety transformation can be used as the basis for VTS tests (cf. Liveness to safety reduction and prior Apalache-related work).
  • Assess whether this is sufficient for the full range of TLA+ liveness features. Known concern: this may not adequately cover more advanced TLA+ liveness properties, especially properties over actions.
  1. Implementation-level assurance for TLC liveness
  • Explore a more technical/formal route focused on establishing correctness of TLC’s liveness-checking implementation itself, rather than relying only on behavioral differential testing (as is done for safety).
  • This would likely require a different qualification argument than the current VTS approach.
  1. Behavioral testing using TLAPS (emerging option)
  • Evaluate whether recent/ongoing TLAPS support for proving liveness properties could enable a behavioral-testing strategy analogous to the current safety validation workflow. If viable, this may provide an oracle or reference basis for liveness test cases.

Outcome needed: a documented validation strategy (or combination of strategies) for qualifying liveness checking in the VTS, with explicit coverage limits and rationale appropriate for safety-critical qualification contexts (e.g., ISO 26262).

/cc @konnov

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the current VTS safety-validation workflow and the three proposed approaches: liveness-to-safety reduction, implementation-level assurance, and TLAPS-based behavioral testing. Done means documenting a validation strategy or combination of strategies, including coverage limits and rationale for safety-critical qualification contexts such as ISO 26262.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.