cucapra / cucapra/protocols

Simplify testing infra / create centralized Python script that orchestrates Turnt

Open
#191 1 comment 0 reactions 0 assignees View on GitHub
Chore
Dominant language
Rust
Stars
15
Forks
0
Avg merge
6h 20m
Merged PRs (30d)
15

Description

Kevin mentioned he doesn't like our current snapshot testing setup, specifically:
1. How we have so many different file formats (`.prot`, `.tx`, `.rt`)
2. How we need to create duplicate versions of the same `.prot` file in order to supply arguments to Turnt (e.g. the same `.prot` file is present in both the interpreter & monitor subdirectories)

As we add more tests / larger benchmarks, it would be nice if our testing infra was simpler (to avoid duplicate versions of the same `.prot` file from being out of sync with each other). Kevin suggested having a centralized Python script that orchestrates Turnt. That is, for each `.prot` file, the Python test orchestrator figures the right inputs for Turnt for all the different styles of tests (monitor test, interpreter test, round-trip tests) and then actually invokes the right Turnt environment somehow.

Contributor guide

No contributing guide indexed for this repository

Research direction

No concrete file paths or tests are named. Start by inventorying the existing .prot, .tx, and .rt snapshot tests and their Turnt environments, then trace how interpreter, monitor, and round-trip tests are invoked. Done means one centralized Python orchestrator can select the needed inputs and run each test style without duplicated .prot files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing, tooling
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.