JS conformance: grow Test262 coverage and mature the harness
- Dominant language
- C#
- Stars
- 154
- Forks
- 4
- Avg merge
- 2h 46m
- Merged PRs (30d)
- 189
Description
**Standing task.** The Test262 harness's own maturity and reach. Supersedes #69, #70, #71, #72, #108, #110, #885, and the coverage half of #249.
Sibling issue: closing the interp↔compiled gap is tracked separately — that's about *fixing* what the harness already measures. This one is about *measuring more, and trusting the measurement*.
## Where we stand
11,384 tests in the committed subset, from 13 folders:
```
test/built-ins/{Array, Boolean, Error, JSON, Math, Number, Object, Promise, RegExp, String}
test/language/expressions/{call, new, property-accessors}
```
| Bucket | interpreted | compiled |
|---|---:|---:|
| Pass | 9,823 | 9,861 |
| Fail | 383 | 353 |
| RuntimeError | 288 | 273 |
| ParseError | 10 | 10 |
| HarnessError | — | 7 |
| Timeout | 1 | 1 |
| **Skipped** | **879** | **879** |
**Official aggregation semantics** (any consumer must compute identically): denominator is all tests **excluding** `Skipped:*` — skips are policy, not capability. Numerator is `Pass` only; every other bucket counts as not-passing. Skips are reported separately, never folded into the percentage.
---
## 1. Grow the subset
The corpus is ~48k tests; we run 11,384. Priority order:
- [ ] `test/language/statements/{function, class, for, ...}` — control flow + declarations
- [ ] `test/language/{arguments-object, rest-parameters}`
- [ ] `test/built-ins/{Function, Symbol, Map, Set, Proxy, Reflect, Date}`
- [ ] `test/built-ins/{TypedArray, ArrayBuffer, DataView}`
- [ ] Modules, async, generators
Each rollout step is one PR: add folder(s) to `config/subset.json`, regen both baselines (`SHARPTS_TEST262_UPDATE_BASELINE=1`), commit the new lines, and **triage the new failures** into "fix now" (own bug-fix PR) or "expected gap" (stays in the baseline with its bucket). Do not add a folder without triaging what it surfaces — an untriaged folder is noise, not coverage.
## 2. Unskip what we're skipping
879 tests are skipped in both modes. The two large, actionable blocks:
- [ ] **Negative tests — 194 skipped** (`Skipped:negative-test-deferred`). Every test with a `negative:` frontmatter block is auto-skipped because we have no mapping from SharpTS's internal exceptions to ES error constructor names. Needs: a translation layer (parser exceptions → `SyntaxError`; `TypeCheckException` case-by-case → `SyntaxError` for early errors, `TypeError` otherwise; runtime SharpTS errors → their ES counterpart), then runner support for `phase: parse|resolution|runtime` + `type: ErrorName`, verifying the expected error fires in the expected phase. Mismatch → `Fail`. **Acceptance:** a hand-picked negative test with a deliberately wrong expected `type` fails loudly.
- [ ] **RegExp feature skips — 674 total**: `regexp-unicode-property-escapes` (516), `regexp-v-flag` (63), `regexp-named-groups` (42), `regexp-match-indices` (21), `regexp-lookbehind` (17), `regexp-duplicate-named-groups` (15). These are genuine unimplemented features, not harness gaps — each is its own feature-work decision. `unicode-property-escapes` alone is 516 tests and is the largest single skip block in the suite.
Small remainder: `tail-call-optimization` (6), `iterator-helpers` (4), `SharedArrayBuffer` (1).
## 2b. Known RegExp semantic divergences (both modes)
- [ ] **Negated shorthand escapes inside a character class** (was #749). `\D`/`\W`/`\S` inside `[...]` pass through to .NET unchanged and diverge from JS in **both** modes:
```js
/[\W]/.test("İ") // JS: true, SharpTS: false (.NET \w over-matches U+0130)
/[\S]/.test(" ") // JS: false, SharpTS: true (.NET \s is narrow, so [\S] over-matches Unicode WhiteSpace)
/[\S]/.test(" ") // JS: false, SharpTS: true
```
#693 added `RewriteEcmaScriptShorthands` (shared by interpreter `SharpTSRegExp` and emitted `$RegExp`) and deliberately scoped it: **positive** shorthands (`\d \w \s`) expand everywhere; **negated** ones expand only *outside* a class. In-class positive forms and negated *classes* with positive shorthands (`[^\w]`) are already correct.
Why it was deferred: a negated shorthand inside a class can't be expressed as a plain character set without class nesting/union-of-negation, which .NET's syntax doesn't offer generally. .NET *does* support class subtraction (`[base-[subtract]]`), so the **sole-element** case is tractable — `[\S]` → `[^]`, `[\W]` → `[^A-Za-z0-9_]` — but the union case isn't clean: `[a\S]` = `a ∪ \S`, and since `\S` already contains `a` that collapses to `\S`; a general `[X\S]` has no simple rewrite.
**Suggested scope:** handle the sole-element case (optionally with a leading `^`) in `RewriteEcmaScriptShorthands`, which fixes the repros above; document the union case as a known edge. Interp: `Runtime/Types/SharpTSRegExp.cs` (`RewriteEcmaScriptShorthands`/`ExpandShorthand`). Compiled: `Compilation/RuntimeEmitter.TSRegExp.cs` (`EmitTSRegExpRewriteShorthands`/`EmitTSRegExpExpandShorthand`) — **must stay BCL-only / standalone**. Also still deferred from #693: `iu`-mode case folding (K U+212A, ſ U+017F folding into `\w`).
## 3. Harness correctness and trust
- [ ] **10 `ParseError`s in `RegExp/CharacterClassEscapes/`** (was #108). All 12 generated tests share one shape: a `buildString({loneCodePoints, ranges})` call from the harness's `regExpUtils.js` enumerating the full Unicode range including `0x10FFFF`. Suspects in order: `includes: [regExpUtils.js]` not assembling (check `Test262HarnessAssembler.cs`); an array-of-2-element-arrays parser ambiguity around `[[`; supplementary-plane code points; the ~1.1M-character result string. **Start by running one directly and reading the actual diagnostic** — it should localize the bug, and all 12 will likely flip together.
- [ ] **18 compiled `HarnessError`s** — currently uninvestigated; these are harness failures, not test failures, so they're pure noise in the baseline.
- [ ] **Timeouts under parallel-regen contention** (was #110). Currently down to 2, from ~150 at the time #110 was filed — so this largely self-resolved and the issue text is stale. Keep the 15 s `timeoutSeconds` and only reopen the investigation if the count climbs again. Recorded history: worker recycling made it *worse* (1046 timeouts vs 692); N=4 workers instead of 6 was worse still; raising the timeout 5 s → 15 s was the fix that stuck. Real remedies if ever needed: tier-up JIT control, a shared ALC across tests within a worker, or CPU affinity.
## 4. Strict-mode variant pairs (deliberate, maintainer-gated — was #885)
Test262's canonical harness runs each **unflagged** test twice: once sloppy, once with a `"use strict"` prologue. We run unflagged tests **once, sloppy only**. Adopting variant pairs would match canonical coverage.
**The codegen floor is already in place.** #882/#884 proactively fixed the strict-mode bugs this would expose — a 495-test force-strict sample went `49 → 40 → 34 → 27 → 8 → 4 → 0` sloppy-Pass→strict-Fail across the sweep (strict dynamic property writes on built-ins persist; strict writes on Date/RegExp/Promise/Error PDS objects persist; strict `delete` honors configurability; strict writes honor non-writable/accessor descriptors; strict indexed writes honor `preventExtensions`; strict symbol-keyed and globalThis-sentinel writes persist; `onlyStrict` tests now actually run strict).
**Open decisions — these are yours, not the implementer's:**
- Go / no-go at all, vs. keeping sloppy-only and relying on `onlyStrict`-flagged tests for strict coverage.
- Baseline key encoding for the second variant (the model is one result per path today; variant pairs make it two).
- Default-on vs. opt-in flag, given run time roughly **doubles**.
## 5. CI integration
- [ ] Test262 does **not** run in CI. Regressions are caught on the next ad-hoc run, not before merge. Scope: a GitHub Actions job after the existing suite; submodule shallow-checkout cached by commit SHA (the corpus is ~300 MB); ≤5 min budget for the committed subset — trim or parallelize if over; per-mode diff uploaded as an artifact; job fails the PR on new regressions **or new passes**, matching local xUnit behavior. **Acceptance:** a deliberately broken PR fails the check with a readable diff in the artifact.
Note the interaction: growing the subset (§1) and enabling strict variants (§4) both push against the CI runtime budget. Sequence accordingly.
## 6. Maintenance cadence
- [ ] **Submodule pin bump** (quarterly) — bump `external/test262` to the latest stable SHA, rebaseline, review the diff. New tests landing as `Pass` is free progress; new failures get triaged fix-now vs expected-gap.
- [ ] **Baseline audit** (after each wide sweep) — `SHARPTS_TEST262_WIDE_SWEEP=1`, cluster failures by bucket + subdirectory. A large uniform cluster is either a skip-list candidate or a single root-cause bug worth filing.
## 7. The baselines are a public contract
`SharpTS.Test262/baselines/*.txt` are parsed at build time by [sharpts-www](https://github.com/nickna/sharpts-www) for its `/conformance` page (the site pins SharpTS as a submodule, so the files are already version-matched to the build the playground runs). That makes them externally consumed, not internal harness state.
- [ ] Document the format in `SharpTS.Test262/README.md`: line format ` `, single `#` header line, closed bucket vocabulary (`Pass`, `Fail`, `RuntimeError`, `ParseError`, `Timeout`, `HarnessError`, `Skipped[:reason]`).
- [ ] Add a **version marker** to the header comment, bumped on any format/vocabulary change, so the website parser fails loudly instead of silently miscounting.
- [ ] Document the aggregation semantics stated at the top of this issue in the README, so every consumer computes the same percentage.
- [ ] *(Optional)* baseline lint in CI — check every non-comment line parses as ` `. Protects against hand-edit typos and harness drift.
**Non-goal:** no JSON/artifact generation in SharpTS CI. The raw committed baselines *are* the interface; aggregation and presentation live in the website build.
## Working agreement
- Regenerate **both** baselines together so interp↔compiled stays diffable against one commit.
- Build `SharpTS.Test262.Worker -c Release` before a Release regen — a config mismatch runs compiled tests in-process and stack-overflows testhost.
- Skip-features and eval bucketing are runner-side; direct worker runs will differ from runner baselines on exactly those, so don't chase that diff.
Contributor guide
Research direction
Treat this as a set of separately scoped tasks rather than one change. For a contained investigation, run one RegExp/CharacterClassEscapes test directly, then inspect Test262HarnessAssembler.cs and the referenced regExpUtils.js assembly path; done means identifying the shared cause and recording a verified baseline change for all affected tests. Other sections require maintainer decisions about strict variants, CI, and baseline formats.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, javascript, typescript
- Domain
- ci-cd, compilers, documentation, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100