sjames / sjames/syscribe

feat(ingest-results): manual/exploratory verification session format (session-log)

Open
#113 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Rust
Stars
7
Forks
1
Avg merge
21m
Merged PRs (30d)
2

Description

Summary

ingest-results/testFunctions: cleanly links automated test output (cargo test --format json, JUnit) to TestCase verification status. There's no equivalent for manual/exploratory verification — a session where a human or an LLM agent runs a sequence of commands (curl requests, MQTT probe clients, CLI invocations) against a live system and checks the actual responses against a TestCase's Gherkin scenarios. That evidence currently has nowhere to go but hand-typed prose in a trailing blockquote, which nothing mechanically checks against the scenarios it claims to satisfy.

Motivation

Verifying a security-sensitive feature (constrained remote command execution: an allowlist, a fail-closed default, role-gated dashboard access, per-device credential scoping) meant a live end-to-end session: starting/restarting a real server with different env configurations, issuing curl requests as different roles, running a throwaway MQTT client authenticated with negotiated device credentials, and reading back SQLite rows to confirm outcomes — none of which is a #[test] function cargo test would run standalone (session state, server restarts, and cross-process MQTT timing don't fit that model cleanly). The only record of that verification, once it was done, was a hand-written "Verified live: ..." sentence — which is exactly the kind of claim that's cheap for an LLM to write whether or not it's true, and there's no way for audit/validate to tell the difference between "an agent asserts this passed" and "an agent has a recorded transcript proving it passed."

Proposed shape

A session-log ingestion format alongside cargo-json/junit:

syscribe ingest-results --format session-log <file>

Input: a JSON record per TestCase scenario exercised —

{
  "testCase": "TC-WEB-011",
  "scenario": "An empty/unset allowlist denies everything",
  "steps": [
    {"cmd": "curl -X POST ... /admin/devices/x/commands -d command=restart_hmi", "expect_status": 400},
    {"cmd": "sqlite3 ... SELECT COUNT(*) FROM remote_commands", "expect_output": "2"}
  ],
  "result": "pass",
  "timestamp": "2026-09-12T08:02:29Z"
}

Same downstream effect as ingest-results --format cargo-json: populates .syscribe/results.json, so matrix/trace/audit can annotate that scenario [pass]/[fail]/[unknown] the same way an automated test's outcome does today — the point isn't a new report, it's giving manual/live verification the same machine-checkable status automated tests already get, instead of leaving it as unverifiable prose.

Acceptance criteria

  • ingest-results --format session-log parses the shape above into .syscribe/results.json
  • matrix/trace annotate a TestCase's scenario with [pass]/[fail] sourced from a session-log the same way they do for cargo-json
  • A TestCase with only prose evidence (no ingested result) is visibly distinguished from one with an ingested pass, in both text and --json output
  • Malformed/missing steps fails ingestion with a clear error rather than silently producing an empty result set

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing ingest-results --format cargo-json behavior and the .syscribe/results.json output described in the issue. Then trace how matrix, trace, and audit consume those results, and compare their text and --json handling. Done means valid session records produce scenario statuses, prose-only evidence is distinguished, and malformed or missing steps fails clearly.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
cli, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.