koala73 / koala73/worldmonitor

perf(consumer-prices): shadow-measure repeated Firecrawl validations on pinned product URLs

Open
#7,185 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
86.6k
Forks
13.1k
Avg merge
8h 4m
Merged PRs (30d)
825

Description

### Feature area

API / Backend

### Description

I'd like to propose a bounded, measurement-only experiment around the existing pinned-product Firecrawl path in `consumer-prices-core`.

The scraper stability design deliberately pins stable product URLs and then sends those URLs directly to Firecrawl on later runs. That makes this a good workload for measuring whether some repeated paid validations are redundant within a bounded freshness window.

The proposed first phase would use SeenRelay (`npm install seenrelay`) only in strict shadow mode on a small subset of pinned product validations:

stored/known price
-> SeenRelay CHECK
-> existing Firecrawl extraction (always runs)
-> SeenRelay OBSERVE with the independently extracted result

Important: no Firecrawl call would be skipped in this phase, no price/output would change, no validator would be relaxed, and no fallback/provider routing would change. SeenRelay failure is fail-open, so the existing Firecrawl path continues unchanged.

I suggest bounding the experiment to 10–20 pinned URLs and two normal scheduled scrape cycles (or an equivalently small sample). The output would be measurement only:

- CHECK status distribution;
- candidate SAME_OBSERVED rate at WorldMonitor-selected freshness windows;
- whether the still-executed Firecrawl result actually agrees with the candidate reused value;
- SeenRelay latency overhead;
- Firecrawl calls/credits that would have been avoidable under each tested freshness policy;
- monetary/latency break-even;
- false-reuse candidates and changed values.

If the measured opportunity is negligible, or consumer-price freshness requirements mean Firecrawl should always be called, the correct result is to remove the instrumentation and report that this workload is not a fit.

If this is useful, I can prepare a focused PR limited to shadow instrumentation and local reporting. I would not propose enabling reuse as part of that PR.

### Problem it solves

The consumer-prices pipeline already does real work to stabilize acquisition: pinned product URLs avoid repeated discovery drift, and existing issues explicitly require bounded retries/provider work.

What is still unknown is whether repeated validation of those stable URLs contains a measurable amount of economically redundant provider work.

Today that question cannot be answered from architecture alone. The proposed shadow pilot measures it against the exact Firecrawl results WorldMonitor would have obtained anyway, without changing production semantics.

This gives a falsifiable answer to a concrete cost/latency question:
"Within a freshness window WorldMonitor would actually accept, how often is a repeated paid extraction unchanged, and is a cheap preflight economically worthwhile?"

A negative result is useful: it establishes that price freshness is too strict or repeated work too sparse for this optimization. A positive result gives measured evidence before any behavior-changing reuse is considered.

### Alternatives considered

1. Firecrawl/provider caching alone: useful, but it measures/provider-controls caching inside one provider rather than cross-run/cross-client reuse evidence at the consuming application boundary.

2. A local application cache: this can work well inside one deployment, but would answer a different question and does not measure whether independently produced freshness evidence is reusable across separate callers/fleets.

3. Enabling reuse immediately: rejected. The proposed phase intentionally keeps every Firecrawl call so candidate reuse can be checked against the provider result before changing behavior.

4. More synthetic benchmarks: rejected as the primary evidence. SeenRelay already has small first-party Firecrawl benchmarks; the point here is to measure WorldMonitor's natural workload.

### Additional context

Relevant existing WorldMonitor design/issue context:

- `consumer-prices-core/plans/scraper-stability.md` intentionally uses stable pinned product URLs and a direct Firecrawl path.
- #6182 explicitly requires retries/provider fallback to remain bounded and says known failures should not be called repeatedly within the same scrape.
- This proposal does not replace Firecrawl/Exa, does not increase scraping, and does not change any retailer/source policy. It only instruments validations WorldMonitor already performs.

SeenRelay:
https://github.com/ovladon/seenrelay

Public client:
`npm install seenrelay`

The client defaults to strict shadow mode when no reuse policy is supplied. It performs CHECK, runs the application's validator, then OBSERVEs the independently obtained result. The original validation is not suppressed.

Contributor guide

Open the contributing guide

Research direction

Start by reading consumer-prices-core/plans/scraper-stability.md and locating the existing pinned-product Firecrawl validation path. Review issue #6182 for bounded retry and provider-fallback constraints. Done means a small, measurement-only shadow experiment with local reporting for the listed metrics, while every Firecrawl call and production result remains unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.