awslabs / awslabs/tsai-protocol
Build structured conformance tests and replace the monolithic checker
- Dominant language
- Python
- Stars
- 9
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
The repository's validation coverage is useful but unevenly represented. Some behaviour has portable JSON vectors, while many more complex cases are mutations or synthetic assertions embedded inside `tools/check.py`. The recent freshness-boundary work made this imbalance visible: freshness now has a clear external case matrix, while several more important verification paths have no reusable conformance vectors.
`tools/check.py` has also accumulated schema checks, document parsing, cryptographic verification, fixture mutation, OpenAPI assertions, integrity checks, and cross-reference checks in one script. This makes individual behaviours harder to locate, review, run, and diagnose.
## Scope
Build a structured test suite with one test file, or coherent group of test files, per protocol area. Preserve the existing `python3 tools/check.py` entry point during migration if needed, but move behavioural assertions into independently runnable tests.
Suggested areas include:
- Credential and SD-JWT VC claim validation.
- Presentation and disclosure processing.
- Holder binding, freshness, nonce, and request binding.
- Persistent `sub`, `dct`, domain freshness, and hostname canonicalisation.
- Canonical and derived `vct` schemas, Type Metadata inheritance, and integrity chains.
- Reputation methodology binding and score semantics.
- Token Status List validation, including `typ`, issuer, `sub`, and blocked entries.
- TA operational-report and HSM-attestation JWS verification.
- OpenAPI contracts and examples.
- Documentation examples, links, and prose-embedded integrity values.
## Portable conformance vectors
Add reusable positive and negative vectors for the protocol behaviours that currently exist only as checker mutations, especially:
- Complete SD-JWT VC presentations and disclosure failures.
- Status-list token substitution and header failures.
- `req` method, URI, and body-digest mismatches.
- Canonical and non-canonical `sub`/`dct` hostnames.
- Metadata-integrity mismatch, unrelated inheritance, and circular inheritance.
- Reputation-methodology integrity and semantic failures.
Keep vector data separate from test-runner logic so implementers in other languages can reuse it.
## Acceptance criteria
- Existing checker behaviour is inventoried and retained before old inline assertions are removed.
- Tests are divided into domain-focused files with clear names and isolated failures.
- Positive and negative protocol vectors are documented and reusable outside Python.
- Tests perform no network access and mutate fixtures only in memory.
- One documented command runs the full suite locally and in CI.
- The legacy checker becomes a small compatibility entry point or is removed after callers migrate.
- The repository checker and the new suite produce equivalent pass/fail results during migration.
This work should be performed separately from the current TSAI v1 protocol-definition pull request.
Contributor guide
Research direction
Start by inventorying the assertions and fixture mutations in tools/check.py, then group them against the protocol areas and vectors named in the issue. Done means independently runnable domain-focused tests, reusable positive and negative data, no network access, equivalent checker results during migration, and one documented local/CI command.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa, tooling
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100