block / block/buzz

Proposal: CI for the formal models (Tamarin + TLC + NIP-PL mutation scripts)

Open
#2,621 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32.7k
Forks
4.3k
Avg merge
1d 13h
Merged PRs (30d)
253

Description

## What's missing

The repo's formal artifacts are treated as load-bearing by the docs, but none
of them run in CI:

| Artifact | Claim it backs | How it's run today |
|---|---|---|
| `docs/spec/MultiTenantAuth.spthy` | docs/multi-tenant-relay.md: S1–S8 "machine-verified green", all 32 lemmas on Tamarin 1.12.0 / Maude 3.5.1 | by hand |
| `crates/buzz-core/src/pairing/NIP-AB.spthy` | NIP-AB.md: pairing protocol security lemmas | by hand (`tamarin-prover --prove …`) |
| `docs/spec/MultiTenantRelay.tla` + `.cfg` | docs/multi-tenant-relay.md: exhaustive TLC run, 16.2M distinct states, ~5m | by hand |
| `docs/spec/GitOnObjectStore.tla` + `.cfg` | docs/git-on-object-storage.md: 8 invariants, mutation-tested | by hand |
| `docs/formal/nip-pl/*.py` (6 scripts) | NOTE.md: lease acceptance, stateful gateway, fixed-payload noninterference + mutations | by hand |

VISION.md's isolation bullet says "proven, not asserted". Right now that's true
at the commit where each proof was last hand-run, and silently decays after:
a PR can edit a `.spthy`/`.tla`/model `.py` with no re-check, and toolchain or
spec rot is invisible until someone re-runs manually.

## Proposal

A `formal.yml` workflow — **no Rust build involved** — with three jobs:

1. **Tamarin** (matrix over the two `.spthy` models): install Tamarin 1.12.0 +
Maude 3.5.1 (the exact versions docs/multi-tenant-relay.md records as
verified-green), sha256-pinned against the digests the upstream GitHub
releases publish; run `--prove --quit-on-warning`; gate on the summary
(batch tamarin exits 0 even on falsified lemmas): no `falsified`/`analysis
incomplete`, and verified-lemma count == `lemma` declarations in the file.
2. **TLC** (matrix over the two `.tla` specs): the documented invocations
against the committed bounded `.cfg` harnesses, tla2tools.jar sha256-pinned.
TLC's exit code is already a correct red/green.
3. **NIP-PL scripts**: run all six in NOTE.md order. Two need a tiny patch to
exit non-zero on failure (`acceptance.py` prints violations but exits 0;
`mutation_test.py` prints detection counts without asserting M1>0, M2>0,
SPEC==0) — I'd fold that into the same PR.

Triggers: nightly `schedule` (drift watch) + `workflow_dispatch` +
`pull_request` path-filtered to `docs/spec/**`, `docs/formal/**`, the NIP-AB
`.spthy`, and the workflow itself — so it costs zero minutes on ordinary PRs.

Cost: one ~15m job (MultiTenantRelay TLC on standard runners; documented ~5m on
8 workers), everything else minutes or seconds, only nightly and on
spec-touching PRs.

## Questions for maintainers

- Is nightly + path-filtered PR the right gating level, or would you rather
start nightly-only and promote to a PR gate once it's proven stable?
- Any appetite for a `just formal` recipe mirroring the workflow for local
runs? (Left out of the first PR to keep the diff minimal.)
- #1664 adds a Tamarin S9 model and #1958 a bounded NIP-RS model — both would
be picked up by the path filter once merged; the S9 `.spthy` under
`docs/spec/` gets proven automatically, and new Python models would just be
added to the third job's list.

## Status

The workflow is written, and the two script exit-code fixes are tested (including forced-failure runs). Per CONTRIBUTING's *"for significant changes, open an issue first to discuss the approach,"* I'm **holding the PR pending your nod** rather than sending a new CI workflow unsolicited — happy to open it immediately if this direction sounds right.

Contributor guide

Open the contributing guide

Research direction

Start with the proposed formal.yml workflow, the documented Tamarin and TLC invocations, and docs/formal/nip-pl/NOTE.md. Run the six NIP-PL scripts and inspect acceptance.py and mutation_test.py forced-failure tests, then verify the workflow covers the listed .spthy, .tla/.cfg, and Python artifacts. Done means pinned toolchains run on the stated triggers and failures cannot pass silently.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.