Jordan-Hall / Jordan-Hall/browser

[P0][EVAL-01] Resettable fixtures and protocol tests

Open
#101 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Programme: #1
Epic: #33

## Objective
Build the evaluation world alongside the product so every agent/browser/transaction feature has reproducible authoritative truth instead of relying on model self-report or dangerous live actions.

## Scope
- Resettable fixtures for merchant/catalog/checkout, auctions, social, mail/calendar, project systems, repositories and native desktop apps.
- Seeded users/accounts/balances/permissions/files/source revisions.
- Fake clock/timezone controls and deterministic source events.
- Authoritative fixture APIs for expected postconditions.
- Golden IPC/connector/AgentSession/provider protocol recordings.
- Captured observation replay with all production write routes disabled.
- Scenario manifest recording browser engine, OS image, model hash/quantization, provider version, connector version, permissions and initial state.

## Evaluation rules
- Fixture truth—not agent narrative—determines externally checkable success.
- Replay can never submit live purchases/bids/messages/deletions.
- Test state must be resettable and isolated per run.

## Acceptance criteria
- [ ] Core fixture families reset to identical seeded state reproducibly.
- [ ] Fixture APIs expose authoritative expected outcomes for evaluators.
- [ ] Production irreversible actions are technically unavailable in replay mode.
- [ ] Golden protocol tests cover malformed/version-drift/startup/cancellation cases.
- [ ] Fake clock supports auction/schedule/expiry/timeout testing deterministically.
- [ ] Every benchmark run records the complete declared environment/configuration.

## Dependencies
- CORE-01
- SEC-01

**First phase:** P0
**Maturity target:** P3
**Owner:** security-evaluation-release

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing dependencies CORE-01 and SEC-01, then break down the Scope and Evaluation rules into fixture, replay, protocol, clock, and environment-recording work. No files or existing test entry points are named; done means reproducible isolated fixtures, authoritative outcomes, disabled irreversible replay actions, deterministic protocol and time tests, and complete run metadata.

Written by the indexing model from the issue text.

Assessment

Domain
devtools, security, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.