nickna / nickna/SharpTS

JS conformance: grow Test262 coverage and mature the harness

Open
#1,280 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
154
Forks
4
Avg merge
2h 46m
Merged PRs (30d)
189

Description

**Standing task.** The Test262 harness's own maturity and reach. Supersedes #69, #70, #71, #72, #108, #110, #885, and the coverage half of #249.

Sibling issue: closing the interp↔compiled gap is tracked separately — that's about *fixing* what the harness already measures. This one is about *measuring more, and trusting the measurement*.

## Where we stand

11,384 tests in the committed subset, from 13 folders:

```
test/built-ins/{Array, Boolean, Error, JSON, Math, Number, Object, Promise, RegExp, String}
test/language/expressions/{call, new, property-accessors}
```

| Bucket | interpreted | compiled |
|---|---:|---:|
| Pass | 9,823 | 9,861 |
| Fail | 383 | 353 |
| RuntimeError | 288 | 273 |
| ParseError | 10 | 10 |
| HarnessError | — | 7 |
| Timeout | 1 | 1 |
| **Skipped** | **879** | **879** |

**Official aggregation semantics** (any consumer must compute identically): denominator is all tests **excluding** `Skipped:*` — skips are policy, not capability. Numerator is `Pass` only; every other bucket counts as not-passing. Skips are reported separately, never folded into the percentage.

---

## 1. Grow the subset

The corpus is ~48k tests; we run 11,384. Priority order:

- [ ] `test/language/statements/{function, class, for, ...}` — control flow + declarations
- [ ] `test/language/{arguments-object, rest-parameters}`
- [ ] `test/built-ins/{Function, Symbol, Map, Set, Proxy, Reflect, Date}`
- [ ] `test/built-ins/{TypedArray, ArrayBuffer, DataView}`
- [ ] Modules, async, generators

Each rollout step is one PR: add folder(s) to `config/subset.json`, regen both baselines (`SHARPTS_TEST262_UPDATE_BASELINE=1`), commit the new lines, and **triage the new failures** into "fix now" (own bug-fix PR) or "expected gap" (stays in the baseline with its bucket). Do not add a folder without triaging what it surfaces — an untriaged folder is noise, not coverage.

## 2. Unskip what we're skipping

879 tests are skipped in both modes. The two large, actionable blocks:

- [ ] **Negative tests — 194 skipped** (`Skipped:negative-test-deferred`). Every test with a `negative:` frontmatter block is auto-skipped because we have no mapping from SharpTS's internal exceptions to ES error constructor names. Needs: a translation layer (parser exceptions → `SyntaxError`; `TypeCheckException` case-by-case → `SyntaxError` for early errors, `TypeError` otherwise; runtime SharpTS errors → their ES counterpart), then runner support for `phase: parse|resolution|runtime` + `type: ErrorName`, verifying the expected error fires in the expected phase. Mismatch → `Fail`. **Acceptance:** a hand-picked negative test with a deliberately wrong expected `type` fails loudly.
- [ ] **RegExp feature skips — 674 total**: `regexp-unicode-property-escapes` (516), `regexp-v-flag` (63), `regexp-named-groups` (42), `regexp-match-indices` (21), `regexp-lookbehind` (17), `regexp-duplicate-named-groups` (15). These are genuine unimplemented features, not harness gaps — each is its own feature-work decision. `unicode-property-escapes` alone is 516 tests and is the largest single skip block in the suite.

Small remainder: `tail-call-optimization` (6), `iterator-helpers` (4), `SharedArrayBuffer` (1).

## 2b. Known RegExp semantic divergences (both modes)

- [ ] **Negated shorthand escapes inside a character class** (was #749). `\D`/`\W`/`\S` inside `[...]` pass through to .NET unchanged and diverge from JS in **both** modes:
```js
/[\W]/.test("İ") // JS: true, SharpTS: false (.NET \w over-matches U+0130)
/[\S]/.test(" ") // JS: false, SharpTS: true (.NET \s is narrow, so [\S] over-matches Unicode WhiteSpace)
/[\S]/.test(" ") // JS: false, SharpTS: true
```
#693 added `RewriteEcmaScriptShorthands` (shared by interpreter `SharpTSRegExp` and emitted `$RegExp`) and deliberately scoped it: **positive** shorthands (`\d \w \s`) expand everywhere; **negated** ones expand only *outside* a class. In-class positive forms and negated *classes* with positive shorthands (`[^\w]`) are already correct.

Why it was deferred: a negated shorthand inside a class can't be expressed as a plain character set without class nesting/union-of-negation, which .NET's syntax doesn't offer generally. .NET *does* support class subtraction (`[base-[subtract]]`), so the **sole-element** case is tractable — `[\S]` → `[^]`, `[\W]` → `[^A-Za-z0-9_]` — but the union case isn't clean: `[a\S]` = `a ∪ \S`, and since `\S` already contains `a` that collapses to `\S`; a general `[X\S]` has no simple rewrite.

**Suggested scope:** handle the sole-element case (optionally with a leading `^`) in `RewriteEcmaScriptShorthands`, which fixes the repros above; document the union case as a known edge. Interp: `Runtime/Types/SharpTSRegExp.cs` (`RewriteEcmaScriptShorthands`/`ExpandShorthand`). Compiled: `Compilation/RuntimeEmitter.TSRegExp.cs` (`EmitTSRegExpRewriteShorthands`/`EmitTSRegExpExpandShorthand`) — **must stay BCL-only / standalone**. Also still deferred from #693: `iu`-mode case folding (K U+212A, ſ U+017F folding into `\w`).

## 3. Harness correctness and trust

- [ ] **10 `ParseError`s in `RegExp/CharacterClassEscapes/`** (was #108). All 12 generated tests share one shape: a `buildString({loneCodePoints, ranges})` call from the harness's `regExpUtils.js` enumerating the full Unicode range including `0x10FFFF`. Suspects in order: `includes: [regExpUtils.js]` not assembling (check `Test262HarnessAssembler.cs`); an array-of-2-element-arrays parser ambiguity around `[[`; supplementary-plane code points; the ~1.1M-character result string. **Start by running one directly and reading the actual diagnostic** — it should localize the bug, and all 12 will likely flip together.
- [ ] **18 compiled `HarnessError`s** — currently uninvestigated; these are harness failures, not test failures, so they're pure noise in the baseline.
- [ ] **Timeouts under parallel-regen contention** (was #110). Currently down to 2, from ~150 at the time #110 was filed — so this largely self-resolved and the issue text is stale. Keep the 15 s `timeoutSeconds` and only reopen the investigation if the count climbs again. Recorded history: worker recycling made it *worse* (1046 timeouts vs 692); N=4 workers instead of 6 was worse still; raising the timeout 5 s → 15 s was the fix that stuck. Real remedies if ever needed: tier-up JIT control, a shared ALC across tests within a worker, or CPU affinity.

## 4. Strict-mode variant pairs (deliberate, maintainer-gated — was #885)

Test262's canonical harness runs each **unflagged** test twice: once sloppy, once with a `"use strict"` prologue. We run unflagged tests **once, sloppy only**. Adopting variant pairs would match canonical coverage.

**The codegen floor is already in place.** #882/#884 proactively fixed the strict-mode bugs this would expose — a 495-test force-strict sample went `49 → 40 → 34 → 27 → 8 → 4 → 0` sloppy-Pass→strict-Fail across the sweep (strict dynamic property writes on built-ins persist; strict writes on Date/RegExp/Promise/Error PDS objects persist; strict `delete` honors configurability; strict writes honor non-writable/accessor descriptors; strict indexed writes honor `preventExtensions`; strict symbol-keyed and globalThis-sentinel writes persist; `onlyStrict` tests now actually run strict).

**Open decisions — these are yours, not the implementer's:**
- Go / no-go at all, vs. keeping sloppy-only and relying on `onlyStrict`-flagged tests for strict coverage.
- Baseline key encoding for the second variant (the model is one result per path today; variant pairs make it two).
- Default-on vs. opt-in flag, given run time roughly **doubles**.

## 5. CI integration

- [ ] Test262 does **not** run in CI. Regressions are caught on the next ad-hoc run, not before merge. Scope: a GitHub Actions job after the existing suite; submodule shallow-checkout cached by commit SHA (the corpus is ~300 MB); ≤5 min budget for the committed subset — trim or parallelize if over; per-mode diff uploaded as an artifact; job fails the PR on new regressions **or new passes**, matching local xUnit behavior. **Acceptance:** a deliberately broken PR fails the check with a readable diff in the artifact.

Note the interaction: growing the subset (§1) and enabling strict variants (§4) both push against the CI runtime budget. Sequence accordingly.

## 6. Maintenance cadence

- [ ] **Submodule pin bump** (quarterly) — bump `external/test262` to the latest stable SHA, rebaseline, review the diff. New tests landing as `Pass` is free progress; new failures get triaged fix-now vs expected-gap.
- [ ] **Baseline audit** (after each wide sweep) — `SHARPTS_TEST262_WIDE_SWEEP=1`, cluster failures by bucket + subdirectory. A large uniform cluster is either a skip-list candidate or a single root-cause bug worth filing.

## 7. The baselines are a public contract

`SharpTS.Test262/baselines/*.txt` are parsed at build time by [sharpts-www](https://github.com/nickna/sharpts-www) for its `/conformance` page (the site pins SharpTS as a submodule, so the files are already version-matched to the build the playground runs). That makes them externally consumed, not internal harness state.

- [ ] Document the format in `SharpTS.Test262/README.md`: line format ` `, single `#` header line, closed bucket vocabulary (`Pass`, `Fail`, `RuntimeError`, `ParseError`, `Timeout`, `HarnessError`, `Skipped[:reason]`).
- [ ] Add a **version marker** to the header comment, bumped on any format/vocabulary change, so the website parser fails loudly instead of silently miscounting.
- [ ] Document the aggregation semantics stated at the top of this issue in the README, so every consumer computes the same percentage.
- [ ] *(Optional)* baseline lint in CI — check every non-comment line parses as ` `. Protects against hand-edit typos and harness drift.

**Non-goal:** no JSON/artifact generation in SharpTS CI. The raw committed baselines *are* the interface; aggregation and presentation live in the website build.

## Working agreement

- Regenerate **both** baselines together so interp↔compiled stays diffable against one commit.
- Build `SharpTS.Test262.Worker -c Release` before a Release regen — a config mismatch runs compiled tests in-process and stack-overflows testhost.
- Skip-features and eval bucketing are runner-side; direct worker runs will differ from runner baselines on exactly those, so don't chase that diff.

Contributor guide

Open the contributing guide

Research direction

Treat this as a set of separately scoped tasks rather than one change. For a contained investigation, run one RegExp/CharacterClassEscapes test directly, then inspect Test262HarnessAssembler.cs and the referenced regExpUtils.js assembly path; done means identifying the shared cause and recording a verified baseline change for all affected tests. Other sections require maintainer decisions about strict variants, CI, and baseline formats.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, javascript, typescript
Domain
ci-cd, compilers, documentation, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.