nickna / nickna/SharpTS

TS conformance: pilot per-node *.types baseline parity

Open
#88 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
C#
Stars
154
Forks
4
Avg merge
2h 46m
Merged PRs (30d)
189

Description

Follow-up to the completed diagnostic-parity campaign in #1281. Reactivated after the original deferral gates were cleared: the pinned `lib.*.d.ts` graph is loaded, the checker exposes expression types through `TypeMap`, source spans are retained, deterministic TypeScript type rendering exists for declaration emit, and the language server now has checker-backed semantic infrastructure.

## Goal

Add a small, trustworthy `*.types` conformance track that compares SharpTS's per-node inferred types with the pinned TypeScript compiler's committed `tests/baselines/reference/*.types` files.

The existing diagnostics track answers “did both compilers report the same `(line, TSnnnn)` errors?” This track answers the complementary question: “when both compilers accept the program, did SharpTS infer the same types?” It should expose silent widening, narrowing, generic-inference, conditional-type, overload-return, and accidental-`any` divergences.

This issue deliberately covers `*.types` only. TypeScript's `*.symbols` baselines exercise a distinct binding/member/alias surface and should be planned separately after the type-query path is proven.

## Evidence and starting point

At the pinned TypeScript v6.0.3 revision (`050880ce59e30b356b686bd3144efe24f875ebc8`):

- The upstream tree contains 14,016 tracked `*.types` baseline files, including harness-configuration variants.
- The committed SharpTS diagnostic subset is now 534/534 `Pass`. Of those 534 tests, 531 have at least one matching upstream `*.types` baseline family by basename; selecting the compatible configured variant remains part of Phase 1.
- All eleven specifically named pilot candidates below are diagnostic `Pass` and have a plain upstream `*.types` baseline at the pin.
- TypeScript's `TypeWriterWalker` visits expressions, identifiers, and declaration names in AST preorder; it skips most type-only nodes, queries `getTypeAtLocation`, and renders with a non-truncating type printer.
- Upstream type baselines interleave source text with observations such as `>expression : inferred type` and an underline line.

Relevant SharpTS seams:

- `tests/conformance/SharpTS.TypeScriptConformance/TypeScriptConformanceRunner.cs` — metadata, virtual multi-file program construction, directive mapping, resolver setup, checker execution, and target-specific baseline selection.
- `src/SharpTS/TypeSystem/TypeMap.cs` — resolved types keyed by expression identity.
- `src/SharpTS/Parsing/SourceDocument.cs` / `SpanTable.cs` — source ownership and spans.
- `src/SharpTS/Parsing/Visitors/AstVisitorBase.cs` — exhaustive AST traversal.
- `src/SharpTS/Declaration/TypeInfoDeclarationRenderer.cs` — deterministic TypeScript syntax rendering, currently declaration-oriented and internal.
- `src/SharpTS/TypeSystem/BindingIndex.cs` — checker-owned token/declaration identities; useful later for `*.symbols`, but not part of this issue.

## Current status (2026-08-29)

Phase 1 is implemented and validated in the current worktree; the change is not yet committed or merged. It adds a shared deterministic resolver for `.errors.txt` and `.types`, a source-backed `TypesBaselineParser`, checked-in fixtures, focused parser/resolver tests, and pinned-corpus integration coverage.

At the pinned revision, all 518 configuration-compatible `.types` baselines among the 534 diagnostic-pass tests resolve and parse successfully. Sixteen tests correctly produce `NoBaseline`: three have no basename-matching `.types` family, and thirteen expose only target/module variants outside the runner's selected configuration. The explicit TypeScript conformance project passes all 384 test methods, including the unchanged 534/534 diagnostic aggregate.

Phases 2–6 remain unimplemented. The existing executable conformance track remains diagnostics-only; Phase 1 intentionally does not produce or compare SharpTS type observations yet.
## Design decisions

1. **Use a fixed pilot, not the whole corpus.** Start with diagnostic-passing tests so type mismatches identify silent semantic differences rather than duplicating known diagnostic failures.
2. **Keep type conformance separate from diagnostic conformance.** Do not change the meaning or closed vocabulary of `baselines/interpreted.txt`, which is an external contract consumed by `sharpts-www`.
3. **Compare structured observations, not raw files.** Parse the upstream baseline into observations and compare those with observations produced from SharpTS nodes. Raw text equality would conflate source echo formatting, underlines, and semantic type differences.
4. **Do not normalize away semantics.** Normalization may cover documented presentation-only equivalents, but must not erase alias preservation, literal widening, union constituents, optionality, overload selection, or `any`/`unknown`/`never` differences.
5. **Make unsupported coverage visible.** Missing node types and unsupported render shapes are measured outcomes, not silently skipped observations.
6. **Build a reusable type-query/display surface.** The production API should be usable by a later general TypeScript hover feature, but adding LSP hover is not part of this issue.

## Implementation plan

### Phase 1 — Parse and resolve upstream `*.types` baselines

- [x] Add a `TypesBaselineParser` that reads file sections and produces ordered observations containing at least virtual filename, source line, source text, occurrence ordinal, expected type text, and optional underline metadata.
- [x] Cover single-file, `@filename` multi-file, repeated identical source text, nested observations on one source line, blank lines, and types containing `:`/`;`/braces.
- [x] Generalize the existing target/configuration-aware baseline resolver so `.errors.txt` and `.types` select the same configured variant instead of developing two naming algorithms.
- [x] Treat a missing upstream `*.types` file as `NoBaseline`, distinct from parser or harness failure.
- [x] Add parser/resolver unit tests using small checked-in fixtures modeled on the pinned upstream format.

### Phase 2 — Expose types for the node categories TypeScript measures

- [ ] Retain the `TypeMap` returned by `TypeChecker.CheckModules` in the conformance program result instead of discarding it.
- [ ] Add a checker-owned query index for source-backed declaration/name tokens that are not expressions. Keep expression storage compatible with existing compiler/interpreter consumers.
- [ ] Populate the query index at the semantic resolution points for:
- variable and parameter declarations/usages;
- function, class, interface, enum, namespace, and type-alias names where SharpTS has a resolved value/type;
- property/member names when the containing access or declaration has a resolved member type;
- type-alias declaration names using the evaluated alias body, matching upstream's useful expansion behavior.
- [ ] Never invent types for parse-recovery or synthetic nodes. Hidden/no-source nodes must remain absent and be reported as coverage gaps when upstream contains an observation.
- [ ] Add focused checker tests proving shadowed names, nested scopes, class value-vs-instance meaning, property accesses, aliases, and recovered programs return the intended source-node type.

### Phase 3 — Add a stable conformance display renderer

- [ ] Extract/generalize `TypeInfoDeclarationRenderer` into a deterministic `TypeInfo` display service shared by declaration emit and conformance. Avoid duplicating two large type switches.
- [ ] Add an explicit conformance display mode for TypeScript-compatible choices such as literal preservation, alias expansion where requested, union/intersection ordering, overload display, anonymous object shapes, type parameters, conditional/mapped/indexed-access types, and recursion/cycle handling.
- [ ] Make formatting culture-invariant and non-truncating.
- [ ] Return a typed `UnsupportedTypeDisplay` result for shapes that cannot yet be represented; do not degrade them to `any` or `ToString()`.
- [ ] Add golden unit tests for every currently supported `TypeInfo` family and targeted tests for known presentation differences from `tsc`.

### Phase 4 — Produce SharpTS observations in compatible order

- [ ] Add a dedicated source-order/preorder walker modeled on the pinned TypeScript `TypeWriterWalker` selection rules: expressions, identifiers, and declaration names; skip type-only nodes except evaluated type-alias names.
- [ ] Use each `ParsedModule`'s `SourceDocument` and `SpanTable` to retain virtual filename, source text, and stable ordering. Parent expressions must precede their contained expressions when upstream does the same.
- [ ] Define an observation match key that survives harmless AST-shape differences while remaining unambiguous: virtual file + source line/span text + occurrence ordinal. Report ambiguous keys as harness errors rather than guessing.
- [ ] Classify differences as `TypeTextMismatch`, `MissingSharpTSObservation`, `ExtraSharpTSObservation`, `UnsupportedSharpTSType`, or `HarnessError`.
- [ ] Provide a readable failure diff and an environment switch analogous to `SHARPTS_TSCONFORMANCE_DUMP_FAILURES` for dumping full expected/actual observations.

### Phase 5 — Add the pilot runner and independent committed baseline

- [ ] Refactor shared metadata/program construction out of `TypeScriptConformanceRunner` so diagnostics and type runners use identical virtual files, directives, libraries, resolver behavior, decorator mode, strictness options, and multi-file roots.
- [ ] Add `TypeScriptTypesConformanceRunner` and a dedicated `config/types-subset.json` containing explicit tests only.
- [ ] Start with a diverse 10–12 test cohort drawn from the current 531-test diagnostic-pass/`*.types`-baseline candidate pool, including:
- `es2019/globalThisTypeIndexAccess.ts`
- `es2021/logicalAssignment/logicalAssignment9.ts`
- `types/conditional/inferTypes1.ts`
- `types/conditional/inferTypesWithExtends2.ts`
- `types/keyof/keyofIntersection.ts`
- `types/literal/literalTypes1.ts`
- `types/typeRelationships/assignmentCompatibility/intersectionIncludingPropFromGlobalAugmentation.ts`
- `types/typeRelationships/subtypesAndSuperTypes/stringLiteralTypeIsSubtypeOfString.ts`
- `types/typeRelationships/subtypesAndSuperTypes/subtypesOfUnion.ts`
- one library-surface test such as `es2017/es2017DateAPIs.ts`
- one bigint test such as `es2020/constructBigint.ts`
- one well-known-symbol/property-access test with a diagnostic `Pass`
- [ ] Commit a separate, versioned `baselines/types.txt` aggregate with a documented vocabulary such as `Pass`, `Mismatch`, `Unsupported`, `NoBaseline`, and `HarnessError`.
- [ ] Give the type baseline its own update switch (for example `SHARPTS_TSTYPES_UPDATE_BASELINE=1`) and regression rules. A `Pass -> anything else` transition must fail; intentional baseline changes must be reviewed rather than absorbed.
- [ ] Do not add the new vocabulary to `baselines/interpreted.txt` or silently change the website's diagnostics aggregation.

### Phase 6 — Triage and document the pilot

- [ ] Run the cohort against the pinned TypeScript revision and publish counts for matches, semantic mismatches, display-only mismatches, missing observations, and unsupported display shapes.
- [ ] For each mismatch, determine whether the first fix belongs in inference/checking, node coverage, or rendering. Add focused core tests before changing the aggregate baseline.
- [ ] Document the `.types` track, configuration, update workflow, bucket meanings, pinned-revision requirement, and the distinction between semantic and display parity in the conformance README.
- [ ] Publish the measured pilot result on this issue and link any coherent semantic follow-ups instead of expanding this issue into a general checker backlog.

## Acceptance criteria

- [ ] The runner parses and compares upstream `.types` baselines for single- and multi-file tests without invoking `tsc` at test time.
- [ ] The pilot cohort is fixed, documented, and uses the same compiler options/resolver world as diagnostic conformance.
- [ ] Expected and actual observations produce actionable structured diffs.
- [ ] Literal widening, union constituents, generic/conditional inference, and accidental `any` differences remain observable.
- [ ] Missing nodes and unsupported render shapes are never counted as passes or silently omitted.
- [ ] `baselines/types.txt` is independent, versioned, deterministic, and regression-gated.
- [ ] Existing diagnostic baseline output and its externally consumed format remain unchanged.
- [ ] Core tests and the explicit TypeScript conformance project pass; no Test262 baseline regression is introduced.
- [ ] The issue closes with a measured pilot report and separately filed follow-ups for semantic clusters worth implementing.

## Validation

```bash
dotnet test tests/SharpTS.Tests/SharpTS.Tests.csproj -c Release
dotnet test tests/conformance/SharpTS.TypeScriptConformance/SharpTS.TypeScriptConformance.csproj -c Release
```

Also verify deterministic output with two consecutive type-baseline generations producing no diff.

## Out of scope

- TypeScript `*.symbols` baselines; file a separate issue after the type pilot establishes the common observation/baseline infrastructure.
- JavaScript `*.js` emit baselines (#87).
- Running `tsc` dynamically or regenerating Microsoft's reference files.
- Full-corpus rollout in the first implementation.
- General TypeScript hover UI in `SharpTS.LanguageServer`; this work should expose reusable APIs for that follow-up.
- Changing TypeScript semantics intentionally documented as SharpTS product differences without an explicit project decision.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.