TS conformance: pilot per-node *.types baseline parity
- Dominant language
- C#
- Stars
- 154
- Forks
- 4
- Avg merge
- 2h 46m
- Merged PRs (30d)
- 189
Description
Follow-up to the completed diagnostic-parity campaign in #1281. Reactivated after the original deferral gates were cleared: the pinned `lib.*.d.ts` graph is loaded, the checker exposes expression types through `TypeMap`, source spans are retained, deterministic TypeScript type rendering exists for declaration emit, and the language server now has checker-backed semantic infrastructure.
## Goal
Add a small, trustworthy `*.types` conformance track that compares SharpTS's per-node inferred types with the pinned TypeScript compiler's committed `tests/baselines/reference/*.types` files.
The existing diagnostics track answers “did both compilers report the same `(line, TSnnnn)` errors?” This track answers the complementary question: “when both compilers accept the program, did SharpTS infer the same types?” It should expose silent widening, narrowing, generic-inference, conditional-type, overload-return, and accidental-`any` divergences.
This issue deliberately covers `*.types` only. TypeScript's `*.symbols` baselines exercise a distinct binding/member/alias surface and should be planned separately after the type-query path is proven.
## Evidence and starting point
At the pinned TypeScript v6.0.3 revision (`050880ce59e30b356b686bd3144efe24f875ebc8`):
- The upstream tree contains 14,016 tracked `*.types` baseline files, including harness-configuration variants.
- The committed SharpTS diagnostic subset is now 534/534 `Pass`. Of those 534 tests, 531 have at least one matching upstream `*.types` baseline family by basename; selecting the compatible configured variant remains part of Phase 1.
- All eleven specifically named pilot candidates below are diagnostic `Pass` and have a plain upstream `*.types` baseline at the pin.
- TypeScript's `TypeWriterWalker` visits expressions, identifiers, and declaration names in AST preorder; it skips most type-only nodes, queries `getTypeAtLocation`, and renders with a non-truncating type printer.
- Upstream type baselines interleave source text with observations such as `>expression : inferred type` and an underline line.
Relevant SharpTS seams:
- `tests/conformance/SharpTS.TypeScriptConformance/TypeScriptConformanceRunner.cs` — metadata, virtual multi-file program construction, directive mapping, resolver setup, checker execution, and target-specific baseline selection.
- `src/SharpTS/TypeSystem/TypeMap.cs` — resolved types keyed by expression identity.
- `src/SharpTS/Parsing/SourceDocument.cs` / `SpanTable.cs` — source ownership and spans.
- `src/SharpTS/Parsing/Visitors/AstVisitorBase.cs` — exhaustive AST traversal.
- `src/SharpTS/Declaration/TypeInfoDeclarationRenderer.cs` — deterministic TypeScript syntax rendering, currently declaration-oriented and internal.
- `src/SharpTS/TypeSystem/BindingIndex.cs` — checker-owned token/declaration identities; useful later for `*.symbols`, but not part of this issue.
## Current status (2026-08-29)
Phase 1 is implemented and validated in the current worktree; the change is not yet committed or merged. It adds a shared deterministic resolver for `.errors.txt` and `.types`, a source-backed `TypesBaselineParser`, checked-in fixtures, focused parser/resolver tests, and pinned-corpus integration coverage.
At the pinned revision, all 518 configuration-compatible `.types` baselines among the 534 diagnostic-pass tests resolve and parse successfully. Sixteen tests correctly produce `NoBaseline`: three have no basename-matching `.types` family, and thirteen expose only target/module variants outside the runner's selected configuration. The explicit TypeScript conformance project passes all 384 test methods, including the unchanged 534/534 diagnostic aggregate.
Phases 2–6 remain unimplemented. The existing executable conformance track remains diagnostics-only; Phase 1 intentionally does not produce or compare SharpTS type observations yet.
## Design decisions
1. **Use a fixed pilot, not the whole corpus.** Start with diagnostic-passing tests so type mismatches identify silent semantic differences rather than duplicating known diagnostic failures.
2. **Keep type conformance separate from diagnostic conformance.** Do not change the meaning or closed vocabulary of `baselines/interpreted.txt`, which is an external contract consumed by `sharpts-www`.
3. **Compare structured observations, not raw files.** Parse the upstream baseline into observations and compare those with observations produced from SharpTS nodes. Raw text equality would conflate source echo formatting, underlines, and semantic type differences.
4. **Do not normalize away semantics.** Normalization may cover documented presentation-only equivalents, but must not erase alias preservation, literal widening, union constituents, optionality, overload selection, or `any`/`unknown`/`never` differences.
5. **Make unsupported coverage visible.** Missing node types and unsupported render shapes are measured outcomes, not silently skipped observations.
6. **Build a reusable type-query/display surface.** The production API should be usable by a later general TypeScript hover feature, but adding LSP hover is not part of this issue.
## Implementation plan
### Phase 1 — Parse and resolve upstream `*.types` baselines
- [x] Add a `TypesBaselineParser` that reads file sections and produces ordered observations containing at least virtual filename, source line, source text, occurrence ordinal, expected type text, and optional underline metadata.
- [x] Cover single-file, `@filename` multi-file, repeated identical source text, nested observations on one source line, blank lines, and types containing `:`/`;`/braces.
- [x] Generalize the existing target/configuration-aware baseline resolver so `.errors.txt` and `.types` select the same configured variant instead of developing two naming algorithms.
- [x] Treat a missing upstream `*.types` file as `NoBaseline`, distinct from parser or harness failure.
- [x] Add parser/resolver unit tests using small checked-in fixtures modeled on the pinned upstream format.
### Phase 2 — Expose types for the node categories TypeScript measures
- [ ] Retain the `TypeMap` returned by `TypeChecker.CheckModules` in the conformance program result instead of discarding it.
- [ ] Add a checker-owned query index for source-backed declaration/name tokens that are not expressions. Keep expression storage compatible with existing compiler/interpreter consumers.
- [ ] Populate the query index at the semantic resolution points for:
- variable and parameter declarations/usages;
- function, class, interface, enum, namespace, and type-alias names where SharpTS has a resolved value/type;
- property/member names when the containing access or declaration has a resolved member type;
- type-alias declaration names using the evaluated alias body, matching upstream's useful expansion behavior.
- [ ] Never invent types for parse-recovery or synthetic nodes. Hidden/no-source nodes must remain absent and be reported as coverage gaps when upstream contains an observation.
- [ ] Add focused checker tests proving shadowed names, nested scopes, class value-vs-instance meaning, property accesses, aliases, and recovered programs return the intended source-node type.
### Phase 3 — Add a stable conformance display renderer
- [ ] Extract/generalize `TypeInfoDeclarationRenderer` into a deterministic `TypeInfo` display service shared by declaration emit and conformance. Avoid duplicating two large type switches.
- [ ] Add an explicit conformance display mode for TypeScript-compatible choices such as literal preservation, alias expansion where requested, union/intersection ordering, overload display, anonymous object shapes, type parameters, conditional/mapped/indexed-access types, and recursion/cycle handling.
- [ ] Make formatting culture-invariant and non-truncating.
- [ ] Return a typed `UnsupportedTypeDisplay` result for shapes that cannot yet be represented; do not degrade them to `any` or `ToString()`.
- [ ] Add golden unit tests for every currently supported `TypeInfo` family and targeted tests for known presentation differences from `tsc`.
### Phase 4 — Produce SharpTS observations in compatible order
- [ ] Add a dedicated source-order/preorder walker modeled on the pinned TypeScript `TypeWriterWalker` selection rules: expressions, identifiers, and declaration names; skip type-only nodes except evaluated type-alias names.
- [ ] Use each `ParsedModule`'s `SourceDocument` and `SpanTable` to retain virtual filename, source text, and stable ordering. Parent expressions must precede their contained expressions when upstream does the same.
- [ ] Define an observation match key that survives harmless AST-shape differences while remaining unambiguous: virtual file + source line/span text + occurrence ordinal. Report ambiguous keys as harness errors rather than guessing.
- [ ] Classify differences as `TypeTextMismatch`, `MissingSharpTSObservation`, `ExtraSharpTSObservation`, `UnsupportedSharpTSType`, or `HarnessError`.
- [ ] Provide a readable failure diff and an environment switch analogous to `SHARPTS_TSCONFORMANCE_DUMP_FAILURES` for dumping full expected/actual observations.
### Phase 5 — Add the pilot runner and independent committed baseline
- [ ] Refactor shared metadata/program construction out of `TypeScriptConformanceRunner` so diagnostics and type runners use identical virtual files, directives, libraries, resolver behavior, decorator mode, strictness options, and multi-file roots.
- [ ] Add `TypeScriptTypesConformanceRunner` and a dedicated `config/types-subset.json` containing explicit tests only.
- [ ] Start with a diverse 10–12 test cohort drawn from the current 531-test diagnostic-pass/`*.types`-baseline candidate pool, including:
- `es2019/globalThisTypeIndexAccess.ts`
- `es2021/logicalAssignment/logicalAssignment9.ts`
- `types/conditional/inferTypes1.ts`
- `types/conditional/inferTypesWithExtends2.ts`
- `types/keyof/keyofIntersection.ts`
- `types/literal/literalTypes1.ts`
- `types/typeRelationships/assignmentCompatibility/intersectionIncludingPropFromGlobalAugmentation.ts`
- `types/typeRelationships/subtypesAndSuperTypes/stringLiteralTypeIsSubtypeOfString.ts`
- `types/typeRelationships/subtypesAndSuperTypes/subtypesOfUnion.ts`
- one library-surface test such as `es2017/es2017DateAPIs.ts`
- one bigint test such as `es2020/constructBigint.ts`
- one well-known-symbol/property-access test with a diagnostic `Pass`
- [ ] Commit a separate, versioned `baselines/types.txt` aggregate with a documented vocabulary such as `Pass`, `Mismatch`, `Unsupported`, `NoBaseline`, and `HarnessError`.
- [ ] Give the type baseline its own update switch (for example `SHARPTS_TSTYPES_UPDATE_BASELINE=1`) and regression rules. A `Pass -> anything else` transition must fail; intentional baseline changes must be reviewed rather than absorbed.
- [ ] Do not add the new vocabulary to `baselines/interpreted.txt` or silently change the website's diagnostics aggregation.
### Phase 6 — Triage and document the pilot
- [ ] Run the cohort against the pinned TypeScript revision and publish counts for matches, semantic mismatches, display-only mismatches, missing observations, and unsupported display shapes.
- [ ] For each mismatch, determine whether the first fix belongs in inference/checking, node coverage, or rendering. Add focused core tests before changing the aggregate baseline.
- [ ] Document the `.types` track, configuration, update workflow, bucket meanings, pinned-revision requirement, and the distinction between semantic and display parity in the conformance README.
- [ ] Publish the measured pilot result on this issue and link any coherent semantic follow-ups instead of expanding this issue into a general checker backlog.
## Acceptance criteria
- [ ] The runner parses and compares upstream `.types` baselines for single- and multi-file tests without invoking `tsc` at test time.
- [ ] The pilot cohort is fixed, documented, and uses the same compiler options/resolver world as diagnostic conformance.
- [ ] Expected and actual observations produce actionable structured diffs.
- [ ] Literal widening, union constituents, generic/conditional inference, and accidental `any` differences remain observable.
- [ ] Missing nodes and unsupported render shapes are never counted as passes or silently omitted.
- [ ] `baselines/types.txt` is independent, versioned, deterministic, and regression-gated.
- [ ] Existing diagnostic baseline output and its externally consumed format remain unchanged.
- [ ] Core tests and the explicit TypeScript conformance project pass; no Test262 baseline regression is introduced.
- [ ] The issue closes with a measured pilot report and separately filed follow-ups for semantic clusters worth implementing.
## Validation
```bash
dotnet test tests/SharpTS.Tests/SharpTS.Tests.csproj -c Release
dotnet test tests/conformance/SharpTS.TypeScriptConformance/SharpTS.TypeScriptConformance.csproj -c Release
```
Also verify deterministic output with two consecutive type-baseline generations producing no diff.
## Out of scope
- TypeScript `*.symbols` baselines; file a separate issue after the type pilot establishes the common observation/baseline infrastructure.
- JavaScript `*.js` emit baselines (#87).
- Running `tsc` dynamically or regenerating Microsoft's reference files.
- Full-corpus rollout in the first implementation.
- General TypeScript hover UI in `SharpTS.LanguageServer`; this work should expose reusable APIs for that follow-up.
- Changing TypeScript semantics intentionally documented as SharpTS product differences without an explicit project decision.
Contributor guide
Assessment
This issue has not been assessed yet.