Parser: complete expression and type-node source-span provenance
- Dominant language
- C#
- Stars
- 154
- Forks
- 4
- Avg merge
- 2h 46m
- Merged PRs (30d)
- 189
Description
Parent epic: #1390. Deferred from #1306.
## Goal
Give every source-backed expression and type node an exact, half-open UTF-16 `SourceSpan`, and preserve or hide that provenance through parser/compiler transforms. This is enabling infrastructure for position-sensitive editor features; it must not become a second syntax tree or change AST record equality.
## Current state
`SourceDocument`/`SpanTable` cover declarations, executable statements, and transformed statement provenance. Tokens already carry exact offsets, so existing lexical navigation works. Broad `Expr` and `TypeNode` coverage remains incomplete, and parts of the type-AST migration still carry string forms plus optional node twins.
## Work
- [ ] Inventory every concrete `Expr`, `TypeNode`, and source-addressable auxiliary node (object/type members, tuple elements, signatures, property keys) and classify it as source-backed, synthesized-with-provenance, or intentionally hidden.
- [ ] Add parser helpers that record a production from its first consumed token through its true final token without duplicating offset arithmetic across productions.
- [ ] Instrument all expression productions, including assignment/conditional/binary/unary chains, calls/new/member/index access, arrows/functions/classes, object/array/template literals, JSX/TSX, assertions/satisfies, dynamic import, optional chaining, and computed keys.
- [ ] Instrument all type productions, including qualified/generic references, function/constructor types, unions/intersections/conditionals, mapped/indexed/keyof/infer types, object members/signatures, tuples, predicates, template-literal types, import types, and heritage/type-argument nodes.
- [ ] Preserve offsets across token rescans and split `>>`/`>>>` generic closers.
- [ ] Audit every transform/lowering that replaces or synthesizes expressions/types. Use `CopySpan` for a source-equivalent replacement and `MarkHidden` for scaffolding; never attribute a synthetic operation to an unrelated nearby expression.
- [ ] Ensure position lookup can select the narrowest node deterministically when parent/child spans share boundaries.
- [ ] Complete required type-node twins rather than manufacturing spans for legacy string-only annotations.
- [ ] Document coverage rules and the contract for future AST node additions.
## Verification
- [ ] Add a reflection/visitor-driven coverage test that walks representative parsed ASTs and fails when a source-backed expression/type node has no span.
- [ ] Assert every recorded span is within its `SourceDocument`, is half-open, and is contained by its source-backed parent where grammar nesting requires containment.
- [ ] Add exact-offset tests for multiline expressions/types, Unicode surrogate pairs, comments/trivia, templates, JSX/TSX, decorators, optional chains, nested generics with split closers, and parser recovery.
- [ ] Add transform tests for destructuring, var hoisting, generator-arrow lifting, nested-function lifting, JSX lowering, async/generator rewrites, and other expression-producing lowerings; synthetic nodes must be hidden or explicitly traceable.
- [ ] Verify multi-file and virtual/embedded stdlib documents never exchange span provenance.
- [ ] Existing debugger sequence-point and LSP navigation suites remain green.
- [ ] Add/extend parser benchmarks and record before/after wall time and allocations on small files plus a representative multi-module project; investigate and document any regression above 10%.
- [ ] Full unit, Test262, and TypeScript conformance validation is green.
## Success criteria
The issue is complete when coverage tests make missing provenance on any supported source-backed expression/type production a regression, transform provenance is explicit, position lookup is deterministic, and the measured cost is acceptable and documented.
## Non-goals
- Emitting expression-level PDB sequence points merely because spans exist.
- Giving ordinary spans to compiler-only scaffolding.
- Implementing an editor feature in this infrastructure issue.
Contributor guide
Research direction
Start with SourceDocument/SpanTable and the parser entry points for Expr and TypeNode, then inventory the concrete nodes and transforms named in the issue. Add the reflection/visitor-driven coverage and exact-offset tests, and verify debugger/LSP, conformance, benchmark, and multi-file provenance checks. Done means supported source-backed nodes have deterministic spans, synthetic nodes are hidden or traceable, and measured cost is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, typescript
- Domain
- compilers, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100