LSP: add semantic tokens for standalone full mode
- Dominant language
- C#
- Stars
- 154
- Forks
- 4
- Avg merge
- 2h 46m
- Merged PRs (30d)
- 189
Description
Parent epic: #1390. Deferred from #1306. This can proceed in parallel with member-navigation work; use #1391 only where token spans are insufficient.
## Goal
Add semantic-token classification for standalone `full` mode using SharpTS parser/checker truth, while avoiding conflicts with `tsserver` in `interop-only` mode.
## Work
- [ ] Define a stable LSP legend for the token kinds SharpTS can distinguish accurately: namespace, type, class, enum, interface, type parameter, parameter, variable, property, enum member, function, method, keyword, string, number, decorator/macro as appropriate.
- [ ] Define modifiers such as declaration, definition, readonly, static, async, abstract, default-library, and deprecated only where the compiler has authoritative data.
- [ ] Classify identifier tokens from checker binding/member identities and syntax-only tokens from the lexer/parser; never infer semantic kinds from capitalization or text alone.
- [ ] Handle TS and TSX tokenization, decorators, type/value facets, namespace members, private names, optional chains, imports/re-exports, JSX tags/attributes, and CLR interop declarations.
- [ ] Encode tokens in deterministic LSP delta form with valid non-overlapping UTF-16 ranges and stable ordering.
- [ ] Implement full-document tokens first; add range or delta support only if measured clients/workspaces justify it.
- [ ] Use versioned analysis caches, observe cancellation, and reject stale computations.
- [ ] Advertise semantic tokens in `full` mode only. `interop-only` must neither advertise nor serve general semantic tokens.
- [ ] Document the legend, mode behavior, and client setup/limitations.
## Verification
- [ ] Golden decoding tests assert exact token text, kind, modifier set, line/column, length, ordering, and delta encoding across representative TS/TSX files.
- [ ] Cover lexical shadowing, merged type/value declarations, imports/re-exports, namespaces, generics, classes/interfaces/enums, properties/methods, decorators, private/static/readonly members, async functions, JSX, and SharpTS .NET interop constructs.
- [ ] Negative tests prove unresolved/malformed tokens receive only defensible syntax classification and never stale semantic classification.
- [ ] Unicode/surrogate-pair, multiline template/comment, line-ending, and parser-recovery positions are exact.
- [ ] Rapid edits, stale versions, cancellation, and cache invalidation do not publish tokens for an obsolete snapshot.
- [ ] Real initialize handshakes advertise the exact legend only in `full`; `interop-only` capability regression tests remain green.
- [ ] If range/delta is implemented, equivalence tests reconstruct the same final token stream as a fresh full request.
- [ ] Benchmark cold and warm token requests on a large TSX file and multi-module project; document latency, allocations, and payload size.
- [ ] Full unit, language-server pack, extension compile, Test262, and TypeScript conformance validation is green.
## Success criteria
A standalone client can render stable, semantically correct highlighting from SharpTS, every encoded token maps to the intended UTF-16 source text, live edits cannot leak stale classifications, and VS Code coexistence mode remains unaffected.
## Non-goals
- Replacing VS Code's TypeScript semantic highlighting.
- Highlighting unsupported semantics by heuristic.
- Requiring delta-token support for the first release.
Contributor guide
Research direction
Start by locating the LSP semantic-token entry point and its SharpTS parser/checker integration; the issue names no files or specific test paths. Read the initialize-handshake behavior and run the language-server validation first. Done means stable UTF-16 full-document tokens in standalone full mode, with stale and interop-only regression cases covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, typescript
- Domain
- compilers, devtools
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100