feat(ingest-results): manual/exploratory verification session format (session-log)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 7
- Forks
- 1
- Avg merge
- 21m
- Merged PRs (30d)
- 2
Description
Summary
ingest-results/testFunctions: cleanly links automated test output (cargo test --format json, JUnit) to TestCase verification status. There's no equivalent for manual/exploratory verification — a session where a human or an LLM agent runs a sequence of commands (curl requests, MQTT probe clients, CLI invocations) against a live system and checks the actual responses against a TestCase's Gherkin scenarios. That evidence currently has nowhere to go but hand-typed prose in a trailing blockquote, which nothing mechanically checks against the scenarios it claims to satisfy.
Motivation
Verifying a security-sensitive feature (constrained remote command execution: an allowlist, a fail-closed default, role-gated dashboard access, per-device credential scoping) meant a live end-to-end session: starting/restarting a real server with different env configurations, issuing curl requests as different roles, running a throwaway MQTT client authenticated with negotiated device credentials, and reading back SQLite rows to confirm outcomes — none of which is a #[test] function cargo test would run standalone (session state, server restarts, and cross-process MQTT timing don't fit that model cleanly). The only record of that verification, once it was done, was a hand-written "Verified live: ..." sentence — which is exactly the kind of claim that's cheap for an LLM to write whether or not it's true, and there's no way for audit/validate to tell the difference between "an agent asserts this passed" and "an agent has a recorded transcript proving it passed."
Proposed shape
A session-log ingestion format alongside cargo-json/junit:
syscribe ingest-results --format session-log <file>
Input: a JSON record per TestCase scenario exercised —
{
"testCase": "TC-WEB-011",
"scenario": "An empty/unset allowlist denies everything",
"steps": [
{"cmd": "curl -X POST ... /admin/devices/x/commands -d command=restart_hmi", "expect_status": 400},
{"cmd": "sqlite3 ... SELECT COUNT(*) FROM remote_commands", "expect_output": "2"}
],
"result": "pass",
"timestamp": "2026-09-12T08:02:29Z"
}
Same downstream effect as ingest-results --format cargo-json: populates .syscribe/results.json, so matrix/trace/audit can annotate that scenario [pass]/[fail]/[unknown] the same way an automated test's outcome does today — the point isn't a new report, it's giving manual/live verification the same machine-checkable status automated tests already get, instead of leaving it as unverifiable prose.
Acceptance criteria
-
ingest-results --format session-logparses the shape above into.syscribe/results.json -
matrix/traceannotate a TestCase's scenario with[pass]/[fail]sourced from a session-log the same way they do forcargo-json - A TestCase with only prose evidence (no ingested result) is visibly distinguished from one with an ingested pass, in both text and
--jsonoutput - Malformed/missing
stepsfails ingestion with a clear error rather than silently producing an empty result set
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the existing ingest-results --format cargo-json behavior and the .syscribe/results.json output described in the issue. Then trace how matrix, trace, and audit consume those results, and compare their text and --json handling. Done means valid session records produce scenario statuses, prose-only evidence is distinguished, and malformed or missing steps fails clearly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- cli, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100