dcramer / dcramer/peated

Expand setup-agent eval coverage and baselines

Open
#1,203 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
103
Forks
18
Avg merge
1h 19m
Merged PRs (30d)
555

Description

Problem

The configured-source setup agent has a narrow schema and deterministic post-model checks, but its live eval set is small: a few synthetic review sites and one price fixture. There is no catalog setup eval, limited hostile/noisy/unsupported HTML coverage, and no recorded success, cost, or latency baseline by model and instruction version.

Relevant code:

  • apps/server/src/scraper/configured/createScraper.eval.test.ts
  • apps/server/src/scraper/configured/suggestion.price.eval.test.ts
  • apps/server/src/scraper/configured/testWebsites.ts
  • apps/server/src/scraper/configured/setupAgent.ts

Desired outcome

The setup agent has a small, production-derived eval matrix that catches regressions without turning deterministic tests into model tests.

Acceptance criteria

  • Add a sanitized catalog fixture and at least one representative noisy or unsupported page for each supported source kind where behavior differs.
  • Include cases with misleading navigation, missing fields, invalid selectors, and content that must not change source kind or bypass validation.
  • Score proposal validity, deterministic preview success, and final extracted fields separately.
  • Record model, instructions version, attempts, latency, and token/cost data when available.
  • Establish a checked-in baseline and an explicit threshold for accepting prompt/model changes.
  • Keep these checks in pnpm evals; deterministic parser and orchestration cases remain mocked in pnpm test.

Non-goals

  • Add multi-agent setup, autonomous activation, or a general web benchmark.
  • Store publisher pages that Peated is not permitted to retain.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading apps/server/src/scraper/configured/createScraper.eval.test.ts, suggestion.price.eval.test.ts, testWebsites.ts, and setupAgent.ts, then run pnpm evals to inspect the current coverage. Add sanitized catalog and noisy or unsupported fixtures for differing source kinds, separate proposal, preview, and extraction scoring, and record the requested model and run metadata. Done means a checked-in baseline with an explicit threshold while deterministic cases remain in pnpm test.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.