Expand setup-agent eval coverage and baselines
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 103
- Forks
- 18
- Avg merge
- 1h 19m
- Merged PRs (30d)
- 555
Description
Problem
The configured-source setup agent has a narrow schema and deterministic post-model checks, but its live eval set is small: a few synthetic review sites and one price fixture. There is no catalog setup eval, limited hostile/noisy/unsupported HTML coverage, and no recorded success, cost, or latency baseline by model and instruction version.
Relevant code:
apps/server/src/scraper/configured/createScraper.eval.test.tsapps/server/src/scraper/configured/suggestion.price.eval.test.tsapps/server/src/scraper/configured/testWebsites.tsapps/server/src/scraper/configured/setupAgent.ts
Desired outcome
The setup agent has a small, production-derived eval matrix that catches regressions without turning deterministic tests into model tests.
Acceptance criteria
- Add a sanitized catalog fixture and at least one representative noisy or unsupported page for each supported source kind where behavior differs.
- Include cases with misleading navigation, missing fields, invalid selectors, and content that must not change source kind or bypass validation.
- Score proposal validity, deterministic preview success, and final extracted fields separately.
- Record model, instructions version, attempts, latency, and token/cost data when available.
- Establish a checked-in baseline and an explicit threshold for accepting prompt/model changes.
- Keep these checks in
pnpm evals; deterministic parser and orchestration cases remain mocked inpnpm test.
Non-goals
- Add multi-agent setup, autonomous activation, or a general web benchmark.
- Store publisher pages that Peated is not permitted to retain.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading apps/server/src/scraper/configured/createScraper.eval.test.ts, suggestion.price.eval.test.ts, testWebsites.ts, and setupAgent.ts, then run pnpm evals to inspect the current coverage. Add sanitized catalog and noisy or unsupported fixtures for differing source kinds, separate proposal, preview, and extraction scoring, and record the requested model and run metadata. Done means a checked-in baseline with an explicit threshold while deterministic cases remain in pnpm test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend, testing
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100