awslabs / awslabs/tsai-protocol

Build structured conformance tests and replace the monolithic checker

Open
#10 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
9
Forks
1
PR merge metrics
No merged PRs in 30d

Description

## Problem

The repository's validation coverage is useful but unevenly represented. Some behaviour has portable JSON vectors, while many more complex cases are mutations or synthetic assertions embedded inside `tools/check.py`. The recent freshness-boundary work made this imbalance visible: freshness now has a clear external case matrix, while several more important verification paths have no reusable conformance vectors.

`tools/check.py` has also accumulated schema checks, document parsing, cryptographic verification, fixture mutation, OpenAPI assertions, integrity checks, and cross-reference checks in one script. This makes individual behaviours harder to locate, review, run, and diagnose.

## Scope

Build a structured test suite with one test file, or coherent group of test files, per protocol area. Preserve the existing `python3 tools/check.py` entry point during migration if needed, but move behavioural assertions into independently runnable tests.

Suggested areas include:

- Credential and SD-JWT VC claim validation.
- Presentation and disclosure processing.
- Holder binding, freshness, nonce, and request binding.
- Persistent `sub`, `dct`, domain freshness, and hostname canonicalisation.
- Canonical and derived `vct` schemas, Type Metadata inheritance, and integrity chains.
- Reputation methodology binding and score semantics.
- Token Status List validation, including `typ`, issuer, `sub`, and blocked entries.
- TA operational-report and HSM-attestation JWS verification.
- OpenAPI contracts and examples.
- Documentation examples, links, and prose-embedded integrity values.

## Portable conformance vectors

Add reusable positive and negative vectors for the protocol behaviours that currently exist only as checker mutations, especially:

- Complete SD-JWT VC presentations and disclosure failures.
- Status-list token substitution and header failures.
- `req` method, URI, and body-digest mismatches.
- Canonical and non-canonical `sub`/`dct` hostnames.
- Metadata-integrity mismatch, unrelated inheritance, and circular inheritance.
- Reputation-methodology integrity and semantic failures.

Keep vector data separate from test-runner logic so implementers in other languages can reuse it.

## Acceptance criteria

- Existing checker behaviour is inventoried and retained before old inline assertions are removed.
- Tests are divided into domain-focused files with clear names and isolated failures.
- Positive and negative protocol vectors are documented and reusable outside Python.
- Tests perform no network access and mutate fixtures only in memory.
- One documented command runs the full suite locally and in CI.
- The legacy checker becomes a small compatibility entry point or is removed after callers migrate.
- The repository checker and the new suite produce equivalent pass/fail results during migration.

This work should be performed separately from the current TSAI v1 protocol-definition pull request.

Contributor guide

Open the contributing guide

Research direction

Start by inventorying the assertions and fixture mutations in tools/check.py, then group them against the protocol areas and vectors named in the issue. Done means independently runnable domain-focused tests, reusable positive and negative data, no network access, equivalent checker results during migration, and one documented local/CI command.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa, tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.