stacklok / stacklok/modelith

Lint: flag invariants and action.preserves not covered by a test

Open
#6 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Go
Stars
32
Forks
5
PR merge metrics
No merged PRs in 30d

Description

Summary

Add a lint/completeness signal that flags every invariant.id (entity-level and model-level) and every action.preserves entry that is not referenced by any test in the repo. This is the durable, on-mission version of "wire testing in from the model": modelith already emits completeness findings (e.g. entities with no invariants, entities no scenario touches). This extends that discipline outward to the model↔test relationship.

Why a checker, not a generator

The tempting version — generate a skipped t.Skip test stub per invariant — is a false green: coverage tools count the stub as present, go test passes, CI is green, and the test asserts nothing. A directory of 40 skipped invariant tests reads as "we have invariant coverage" while proving nothing. Worse, the model↔test link is by invariant.id, and renaming an invariant is a first-class modelith operation (IDs sharpen as understanding does) — a generator would then clobber human-written bodies or orphan them.

Checking references instead of generating code turns the false-green inside out: rather than emitting stubs that lie, flag the invariants that lack a real test. It's deterministic, its output is disposable (a finding, not owned code), and it survives renames because it checks references rather than owning generated files.

Open design question (must resolve before building)

modelith today reads a single YAML file; traceability requires knowing what tests exist. Two candidate input surfaces:

  1. Marker convention — tests reference the invariant id in a structured, greppable way (e.g. a comment token or test-name convention); modelith scans a configured test path.
  2. Sidecar manifest — a small mapping file (invariant-id → test reference) that the #5 skill emits and the engineer maintains; modelith validates coverage against it.

Option 2 keeps modelith's "reads declared inputs" character intact and pairs naturally with #5; option 1 is lower-friction but expands modelith into a repo scanner. Decide this first — it's the crux of whether this stays on-mission.

Scope

  • New completeness (or a new "traceability") category finding.
  • Likely warning severity by default (an unproven invariant is a gap, not a structural error), with an opt-in strict mode consistent with how completeness is already treated.
  • Golden/lint-model test coverage consistent with the existing internal/lint conventions.

Relationship to the other issues

  • Consumes the traceability convention produced by #5.
  • Replaces the discarded "generate skipped test scaffolds" idea (former Tier 2).

Background

Descends from "Tier 2" of a three-tier design discussion, reshaped by a skeptic pass from generate stubs to check references. Consider recording the decision in audits/.

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

First resolve the marker-convention versus sidecar-manifest question, including how it relates to issue #5 and the single YAML input. Then read the existing internal/lint conventions and golden/lint-model tests, and record the decision in audits/ if appropriate. Done means a completeness or traceability finding identifies unreferenced invariant.id and action.preserves entries, with warning-by-default and strict-mode behavior covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
devtools, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.