Jordan-Hall / Jordan-Hall/browser
[P0][EVAL-01] Resettable fixtures and protocol tests
- Dominant language
- No language data
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Programme: #1
Epic: #33
## Objective
Build the evaluation world alongside the product so every agent/browser/transaction feature has reproducible authoritative truth instead of relying on model self-report or dangerous live actions.
## Scope
- Resettable fixtures for merchant/catalog/checkout, auctions, social, mail/calendar, project systems, repositories and native desktop apps.
- Seeded users/accounts/balances/permissions/files/source revisions.
- Fake clock/timezone controls and deterministic source events.
- Authoritative fixture APIs for expected postconditions.
- Golden IPC/connector/AgentSession/provider protocol recordings.
- Captured observation replay with all production write routes disabled.
- Scenario manifest recording browser engine, OS image, model hash/quantization, provider version, connector version, permissions and initial state.
## Evaluation rules
- Fixture truth—not agent narrative—determines externally checkable success.
- Replay can never submit live purchases/bids/messages/deletions.
- Test state must be resettable and isolated per run.
## Acceptance criteria
- [ ] Core fixture families reset to identical seeded state reproducibly.
- [ ] Fixture APIs expose authoritative expected outcomes for evaluators.
- [ ] Production irreversible actions are technically unavailable in replay mode.
- [ ] Golden protocol tests cover malformed/version-drift/startup/cancellation cases.
- [ ] Fake clock supports auction/schedule/expiry/timeout testing deterministically.
- [ ] Every benchmark run records the complete declared environment/configuration.
## Dependencies
- CORE-01
- SEC-01
**First phase:** P0
**Maturity target:** P3
**Owner:** security-evaluation-release
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing dependencies CORE-01 and SEC-01, then break down the Scope and Evaluation rules into fixture, replay, protocol, clock, and environment-recording work. No files or existing test entry points are named; done means reproducible isolated fixtures, authoritative outcomes, disabled irreversible replay actions, deterministic protocol and time tests, and complete run metadata.
Written by the indexing model from the issue text.
Assessment
- Domain
- devtools, security, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100