dotnet / dotnet/runtime

[browser][coreCLR] R2R: Measurement harness — JSON steady-state, correctness, and R2R coverage

Open
#130,520 1 comment 0 reactions 1 assignee Claimed by @pavelsavara View on GitHub
arch-wasm area-Tracing-coreclr os-browser
Dominant language
C#
Stars
18.3k
Forks
5.6k
PR merge metrics
PR metrics pending

Description

# Measurement harness: JSON steady-state, correctness, and R2R coverage

Part of #130524.

Status: **draft / issue candidate** · Area: CoreCLR-on-WASM, measurement, diagnostics/EventPipe · Related: #130518 (PGO trace capture), #130522/#130516 (entry-point upgrades)

## Goal

The measurement harness for the prototype: a **JSON steady-state benchmark**, a **correctness gate across execution modes**, and an **R2R coverage** signal. Coverage — which methods actually executed as R2R native rather than interpreted — is obtained by consuming the runtime's **existing per-method entry-point event over EventPipe on WASM**, at the same scope and granularity as the existing PGO EventPipe trace. Together these are the prototype's success gate (correctness plus steady-state JSON speedup) and the acceptance signal for the lazy tier-up loop.

## Why

The prototype's success bar (correctness plus steady-state JSON speedup) needs proof that the speedup came from code running native, and the lazy loop (#130516 trigger, #130522 entry-point patch) needs an observable confirmation that shards loaded and entry points upgraded. Rather than invent a bespoke counter, reuse the per-method entry-point eventing already flowing through EventPipe — the same trace mechanism used to capture the PGO profile — so coverage is just another view of that trace.

## Direction

- **JSON steady-state benchmark.** Drive a representative JSON parse/serialize workload and measure steady-state throughput, so the R2R speedup is measured on the intended hot path.
- **Correctness across execution modes.** Assert identical results across interpreter-only, startup-R2R, and after-lazy-tier-up runs — splitting, precompiling, and streaming must not change behavior.
- **Report timing, don't over-gate it.** Report steady-state throughput (the success signal) and keep startup timing informational for this prototype.
- **Consume the existing per-method entry-point event on WASM**, using the same EventPipe trace path as the profile capture (#130518). Coverage granularity is whatever that trace already provides (per-method entry-point resolution).
- **Produce a coverage view for the harness:** the set of methods that resolved to R2R native, plus an aggregate derived from it — enough to confirm the JSON hot path ran native and to quantify overall coverage.
- **Confirm the event fires on WASM.** Verify the per-method entry-point event is actually emitted on WASM, including for the entry-point upgrades that #130522/#130516 perform; if it isn't, scope the minimal emission needed.
- **Measurement-time capability.** Coverage is collected in a traced run; timing runs can be separate so tracing overhead does not distort the speedup numbers.

## Scope

**In:** the JSON steady-state benchmark; the correctness gate across execution modes; steady-state timing; and coverage — enabling/consuming the existing per-method entry-point event over EventPipe on WASM to produce a per-method-plus-aggregate coverage view, verifying the event fires on WASM.

**Out:** a new bespoke coverage counter (rejected in favor of the existing event); the profile-production trigger (#130518); the runtime features under test (the other R1 work items).

## Relationship to other items

Shares the EventPipe trace mechanism with the profile capture (#130518) — same transport, different events. It observes the entry-point upgrades produced by #130522 and #130516, so it doubles as the acceptance signal that the lazy loop actually activated shards.

## Success criteria

- From a traced WASM run, the harness obtains the set of methods that executed as R2R native, confirms the JSON hot path among them, and reports an aggregate coverage figure.
- Coverage collection is separable from the timing run so it does not distort speedup numbers.

## Open questions

- Whether the per-method entry-point event actually fires on WASM today, and if not, what minimal emission is needed for the #130522/#130516 upgrade path.
- How to collect the EventPipe trace in the browser for this purpose (shares the browser-capture mechanics of #130518).
- Overhead of tracing during a measured run, and keeping timing and coverage runs cleanly separated.

> [!NOTE]
> This issue was drafted with GitHub Copilot assistance.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.