nickna / nickna/SharpTS

LSP: add semantic tokens for standalone full mode

Open
#1,396 0 comments 0 reactions 0 assignees View on GitHub
deferred enhancement
Dominant language
C#
Stars
154
Forks
4
Avg merge
2h 46m
Merged PRs (30d)
189

Description

Parent epic: #1390. Deferred from #1306. This can proceed in parallel with member-navigation work; use #1391 only where token spans are insufficient.

## Goal

Add semantic-token classification for standalone `full` mode using SharpTS parser/checker truth, while avoiding conflicts with `tsserver` in `interop-only` mode.

## Work

- [ ] Define a stable LSP legend for the token kinds SharpTS can distinguish accurately: namespace, type, class, enum, interface, type parameter, parameter, variable, property, enum member, function, method, keyword, string, number, decorator/macro as appropriate.
- [ ] Define modifiers such as declaration, definition, readonly, static, async, abstract, default-library, and deprecated only where the compiler has authoritative data.
- [ ] Classify identifier tokens from checker binding/member identities and syntax-only tokens from the lexer/parser; never infer semantic kinds from capitalization or text alone.
- [ ] Handle TS and TSX tokenization, decorators, type/value facets, namespace members, private names, optional chains, imports/re-exports, JSX tags/attributes, and CLR interop declarations.
- [ ] Encode tokens in deterministic LSP delta form with valid non-overlapping UTF-16 ranges and stable ordering.
- [ ] Implement full-document tokens first; add range or delta support only if measured clients/workspaces justify it.
- [ ] Use versioned analysis caches, observe cancellation, and reject stale computations.
- [ ] Advertise semantic tokens in `full` mode only. `interop-only` must neither advertise nor serve general semantic tokens.
- [ ] Document the legend, mode behavior, and client setup/limitations.

## Verification

- [ ] Golden decoding tests assert exact token text, kind, modifier set, line/column, length, ordering, and delta encoding across representative TS/TSX files.
- [ ] Cover lexical shadowing, merged type/value declarations, imports/re-exports, namespaces, generics, classes/interfaces/enums, properties/methods, decorators, private/static/readonly members, async functions, JSX, and SharpTS .NET interop constructs.
- [ ] Negative tests prove unresolved/malformed tokens receive only defensible syntax classification and never stale semantic classification.
- [ ] Unicode/surrogate-pair, multiline template/comment, line-ending, and parser-recovery positions are exact.
- [ ] Rapid edits, stale versions, cancellation, and cache invalidation do not publish tokens for an obsolete snapshot.
- [ ] Real initialize handshakes advertise the exact legend only in `full`; `interop-only` capability regression tests remain green.
- [ ] If range/delta is implemented, equivalence tests reconstruct the same final token stream as a fresh full request.
- [ ] Benchmark cold and warm token requests on a large TSX file and multi-module project; document latency, allocations, and payload size.
- [ ] Full unit, language-server pack, extension compile, Test262, and TypeScript conformance validation is green.

## Success criteria

A standalone client can render stable, semantically correct highlighting from SharpTS, every encoded token maps to the intended UTF-16 source text, live edits cannot leak stale classifications, and VS Code coexistence mode remains unaffected.

## Non-goals

- Replacing VS Code's TypeScript semantic highlighting.
- Highlighting unsupported semantics by heuristic.
- Requiring delta-token support for the first release.

Contributor guide

Open the contributing guide

Research direction

Start by locating the LSP semantic-token entry point and its SharpTS parser/checker integration; the issue names no files or specific test paths. Read the initialize-handshake behavior and run the language-server validation first. Done means stable UTF-16 full-document tokens in standalone full mode, with stale and interop-only regression cases covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp, typescript
Domain
compilers, devtools
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.