Semantic tokens are too broad
- Dominant language
- TypeScript
- Stars
- 3.1k
- Forks
- 737
- Avg merge
- 17h 56m
- Merged PRs (30d)
- 21
Description
## Is your feature request related to a problem? Please describe.
C# semantic tokens are currently too broad, which makes theme authors lose distinctions that are already available from the TextMate grammar.
For example, predefined type keywords such as `string` are classified as plain semantic `keyword`. That makes them indistinguishable from access/modifier keywords like `public` and `const` in `semanticTokenColors`.
TextMate has the more precise `keyword.type.cs`, but semantic highlighting overrides it, so themes cannot keep C# type keywords in an intended type color while leaving normal keywords in a keyword color.
The same happens with strings. The semantic token is `string` and captures the whole string without any distinction between punctuation (`"`) and the actual string content, while multiple themes (and my personal preference) color `"` slightly differently. Not sure why the semantic token `string` is even needed here, since strings are usually well detectable by normal TextMate grammar.
semanticHighlighting disabled:
semanticHighlighting enabled:
.
semanticHighlighting disabled:
semanticHighlighting enabled:
Notice the tradeoff: we either have the desired colors for types or string punctuation, or proper semantics for used class names and other types.
In general, TextMate theme authors invest a lot of time to properly color different types of tokens by their semantic meaning. Semantic tokens are also essential for conveying semantics that a TextMate grammar cannot know, but overly broad semantic tokens hurt because more precise TextMate distinctions are effectively lost.
Disabling semantic tokens is often not an option, as you can see from the examples above.
## Describe the solution you would like
The solution, beyond implementing this on the VS Code side (https://github.com/microsoft/vscode/issues/165788), would be to add more precise semantic token classifications, modifiers, or C#-specific semantic token subtypes for cases where the current semantic token is broader than the TextMate scope it replaces.
Examples:
- C# predefined type aliases such as `string`, `int`, `bool`, etc. should not be only `keyword:csharp`; possible alternatives could be `type.defaultLibrary:csharp`, `type:csharp`, or a C#-specific semantic subtype/modifier that themes can target separately from normal keywords.
- String punctuation should remain distinguishable, either by not overriding precise TextMate scopes with broad `string` tokens or by emitting specific semantic token subtypes/modifiers for it.
Semantic tokens should preserve or improve the precision available to themes. They should not carpet-replace precise TextMate classification with a broader semantic one unless there is another way for theme authors to recover that distinction.
## Applicable Scenarios
Users and theme authors who need semantic highlighting to stay enabled but still need tokens with different meanings to remain distinct colors.
## Describe alternatives you've considered
- Disabling semantic highlighting for C# works around this, but it throws away useful semantic coloring entirely.
- Styling `keyword:csharp` is too broad because it also changes normal C# keywords.
- Styling `string:csharp` is too broad because it also changes the whole string, not only string punctuation.
- VS Code feature request microsoft/vscode#165788 from 2022 would also help by allowing semantic token rules to use TextMate scopes as an extra condition, but that is closed as not planned because it did not gain 20 upvotes. Given the current VS Code theming model, the C# extension can preserve theme control by emitting semantic tokens that are at least as precise as the TextMate scopes they override.
## Additional context
Related older issue: dotnet/vscode-csharp#5587. That issue was closed as not planned and labeled OmniSharp, but the same general problem is still visible with the current C# extension / Roslyn LSP path.
Environment where I observed this:
```text
VS Code: 1.116.0
C# extension: 2.140.9
C# Dev Kit: 3.20.199
Roslyn server listed by extension package: 5.7.0-1.26220.12
Using OmniSharp: false / not configured
.NET SDK: 8.0.420
OS: Windows 11, 10.0.26100, win-x64
Theme: any that colors C# types/keywords or string delimiters/content differently
```
Contributor guide
Research direction
No files or tests are identified in the issue. Start by tracing semantic token generation in the current C# extension and Roslyn LSP path, then compare emitted tokens with the TextMate scopes they override. Done means C# predefined types and string punctuation retain distinct, themeable classifications without losing useful semantic highlighting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp, typescript, vscode
- Domain
- developer-experience, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100