HarperFast / HarperFast/prerender-plugin
Measure k: the per-page script-driven origin calls the offload figure counts on neither side
- Vorherrschende Sprache
- JavaScript
- Sterne
- 0
- Forks
- 0
- Ø Merge
- 9 Std. 39 Min.
- Gemergte PRs (30 T.)
- 39
Beschreibung
Follow-up to #152. The console states net origin offload **documents-only** and says the fifth term is
missing from **both** sides of the ledger: the XHR/API calls a page's own scripts make when a rendering
crawler runs it. This issue measures that term so the console can count it instead of stating it.
The caveat is still in the tree (`packages/console/README.md`): *"None of it passes through the plugin,
so the figure is documents-only on both sides … the true net offload for rendering crawlers is higher
than shown."*
## Status — updated 2026-09-16
**Not shipped.** #154 implemented all three stages and was **closed unmerged**; nothing from it is on
`main` (no `hydration_calls`, no `uncacheableSubrequests`, no `scriptsStripped`, no `rendersJs`). The
work below stands as designed; only the version plan needed renumbering, because the tags it reserved
were consumed by the #155–#159 release train.
| stage | was | now |
|---|---|---|
| browser | v1.22.0 | **v1.25.0** (`main` is 1.24.0) |
| plugin | v0.65.0 | **v0.68.0** (0.66.1 shipped #106; 0.67.0 is taken by #102's PR #162) |
| console | v0.13.0 | **v0.14.0** (`main` is 0.13.0 — went to #159) |
One interaction to fold in before re-implementing: #158's endpoint switch drops the page's own
price/availability XHR from bot renders via a fleet `block.urlPatterns` entry. That shrinks the
renderer's own `k` on that deployment and raises the `blocked` count — which the design already
reports, and which is exactly why `blocked` is counted separately rather than silently lost.
## The quantity
`k(page)` = same-origin subrequests a page load makes whose response **no shared cache would serve** —
the calls that reach the origin whoever runs the page. Then, per page-view by a crawler that executes
scripts:
| | origin requests |
|---|---|
| without prerender | `1 + k` (document, then the page's calls) |
| cache-served, snapshot **without** scripts | `0` — `k` is *saved* |
| cache-served, snapshot **with** scripts | `k` — *incurred* |
| proxied origin page | `1 + k` — *incurred* |
| our own render | `1 + k` — *incurred* (the renderer runs the page too) |
## Where each piece is measured
**Browser (`v1.25.0`)** — the renderer's response hook already sees every same-origin response and
already inspects cache headers for its own resource cache. Add a pure classifier over
`(request, response)` → `uncacheable` (explicit: non-GET, `no-store`/`private`/`no-cache`,
`Set-Cookie`, uncacheable status), `cacheable` (explicit positive freshness: `s-maxage`/`max-age`/
`Expires`), or `unspecified` (no freshness info — CDN-default dependent; reported, counted on neither
side). Also count same-origin requests our block list aborted (`blocked`), so the undercount is
visible. Carry the counts on the attempt and post them with the result as
`subrequests: { sameOrigin, cacheable, uncacheable, unspecified, blocked }`, plus
`scriptsStripped: ` so the plugin knows whether the stored snapshot can
hydrate at all. Cost: string checks per response; ~5 ints on the wire.
**Plugin (`prerender-v0.68.0`)**
- `PrerenderedPage` gains `uncacheableSubrequests: Int` and `scriptsStripped: Boolean` (nullable; older
rows read as unknown). Written on store.
- `render` gains series `subrequests` (method = kind) — a value per posted result, so Σ = mean × count;
this is the render fleet's own `k` cost.
- Registry: `analytics.bots[].rendersJs` (boolean). Defaults `true` for the documented renderers only —
Googlebot, Google InspectionTool, Bingbot, Applebot, YandexBot. Every AI crawler in the registry runs
nothing and stays unflagged.
- Serve path: for a request from a flagged crawler, emit `hydration_calls` (new metric; it needs its own
three slots): path = side (`saved` | `incurred` | `unknown`), method = bot, type = source, value = `k`.
`saved` = cache serve of a script-stripped snapshot; `incurred` = cache serve of a snapshot with
scripts, or any origin serve; `unknown` = no page record, or a page that predates the upgrade. One
in-memory counter bump; ~30 combo rows/period/node.
**Console (`prerender-console-v0.14.0`)** — `originLoad()` becomes
`net = 1 − (proxied + renders + Σk_renders + probes + sitemaps + Σk_incurred) ÷ (arrived + Σk_saved + Σk_incurred)`,
the exposure tile becomes "script calls: saved X · incurred Y", the `unknown` count is printed (it decays
over one render cycle after the plugin deploys), and `unspecified` is shown as the CDN-default caveat.
## Order
Every stage tolerates the others being absent (the console falls back to v0.12.0's exposure count; the
plugin ignores fields an older fleet does not post). Browser first so `k` starts landing on pages;
plugin; console. Plugin + console land in one PR here (the console's catalog guard scans the plugin's
emit sites, so a plugin-only merge would fail the console suite), released as two tags.
## Not resource-intensive
Measurement rides on hooks that already run per response; storage is two small nullable fields per page;
the serve path adds one counter bump for rendering-crawler serves. The cost is coordination and the
one-render-cycle warm-up.
Beitragsleitfaden
Rechercherichtung
Start with the caveat in packages/console/README.md, then trace the browser response hook, postProcess.stripScripts, the plugin serve path, and console originLoad() described here. Done means the browser, plugin, and console stages tolerate missing peers, report saved/incurred/unknown and unspecified values, and the console suite passes with the plugin catalog guard.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- javascript
- Bereich
- analytics, full-stack
- Issue-Typ
- Feature
- Schwierigkeit
- 5/5
- Geschätzter Aufwand
- Über eine Woche
- Aktivitätsstatus
- Aktiv
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 48/100