Inconsistencies, Bugs and Generally Confusing Documentation/Guides for Theming and Syntax-Highlighting

Does this issue occur when all extensions are disabled?: Yes/No

unrelated

• VS Code Version: all
• OS Version: all

Steps to Reproduce:

1. Try to learn how to build your own custom Theme
2. Spend hours and hours on research trial and error and end up confused and somewhat frustrated...

I'm not 100% sure if my issue is really a bug report - in parts it's also a feature request, contains questions and even has elements of a tutorial.... sorry..

Background disclosure: I'm an engineer/developer and designer (architect by education) and have decades of experience in both fields. Which is worth nothing if you don't keep learning new things every day. ..

So I decided to build myself a theme for my favorite editor.

Let me share my experience:

Getting Started at Color Theme

I stared with: https://code.visualstudio.com/api/extension-guides/color-theme
Which seems a logical point to start. But even as the page is titled 'Color Theme' the first half of the document is about changing settings.json to modify your currently active them... (It might just be dumb old my, but it took me quite a while to understand that this was totally not what I was looking for.)

Then 'Create a new Color theme' started with the sentence

"Once you have tweaked your theme colors using workbench.colorCustomizations and editor.tokenColorCustomizations, it's time to create the actual theme."

So I had to tweak my theme before even creating it? I felt like in an old Start-Trek episode caught in a logical time-loop.

After several days of research and experimenting I know what the intention of the tutorial is. Actually not to start a theme from scratch but to modify an existing theme. But as that is actually not explained anywhere, that is totally confusing. A series of mostly unexplained steps follow, including a reference to tmThemes (I absolutely had no idea what that 'TextMate' theme was and why I would want/need it...) a link to ColorSublime didn't make anything clearer - to the contrary.

After finding this video: https://www.youtube.com/watch?v=FeApSxfazVg that also is somewhere to be found on the docs, but not linked on the afore mentioned page (why?). The video helped, but as video tutorials go they mostly lack (technical) depth and the syntax highlining, which I was especially interested in and where the TextMate theme thing would have been really helpful to understand, was skipped over quickly. (I'll explain later why I feel that's a bad thing.)

So i stared from scratch in Yeoman. And what I got, while a good starting point but very different from the situation described in the 'Color Theme' page. By the way why does the link behind 'Yeoman' lead to yeoman.io and there is no link to Your First Extension. (May be my fault that I wasn't fully aware that a theme is an extension, when I stared out on the journey, but I strongly feel that a general Explanation of the concept of extensions and themes as one use case should be added/linked)

I don't want this to sound like a rant, and I will not go on in detail like I did until now, but you might trust me that the experience in the next steps was mostly similar.

Workbench Colors

The most valuable resource to have is https://code.visualstudio.com/api/references/theme-color which I found only after my second go around with 'Creating a VS Code Theme' on CSS-Tricks which is linked at the very end of the page I started from, and isn't bad, but full of things I wasn't interned in and also lacking some detail.
The 'Theme Color Reference' has two fundamental flaws:
-no documentation on inheritance.
-[almost] no documentation on where to find the affected element.

There is one Element, that even has two own issues panelSectionHeader.border #1 2
It still took me 2 hours to figure it out. (Can you?)
Another trick one was the listFilterWidget where I found the page of the January 2019 update as a reference, it just wouldn't work as described and when I finally found it it looked an felt totally different.
Sidenote: What I find bothersome is that I didn't know about this useful feature (now I know!) and it took me an hour to find about that as it seems there is either no mention in the docs or unter a totally different name. (I actually found now it under 'Get Started-> User Interface ') but the problem ist that there is no connection anyone can make automatically between listFIlterWidget and advanced tree navigation adding that to the list would at leas help a bit.
I also found a page that attempted to solve that issue https://sw27.net/vsc/color-theme-guide/ but it's incomplete and a bit outdated, and linked nowhere.

Syntax Highlight Guide

This guide is linked the Color Theme page. The page is correctly (as I know by now) sorted under 'Language Extensions' and contains exactly one paragraph about 'Theming'.

I think that is a severe oversight! syntax highlighting is an essential part of theme creation, in my mind probably the most important. I had to research around the web what 'regular TextMate' themes means, nowhere is the use of plists explains even though they are used in the example https://github.com/microsoft/vscode-extension-samples/tree/main/theme-sample which is also linked nowhere in the docs... Also there is no mention I could see about the lack of support for background in vs-code's version of the TextMate themes. Only after hours of trying I found this old Issue.

...

Semantic Highlight Guide

This is one cool feature and a lot of thought and work has gone into it. Using it for themes though is hard.
It starts with the same problem as the syntax highlighting guide from my viewpoint as a would be theme developer: It is written to at least 80% for language extension developers.

The KEY INFORMATION about semantic highlighting for theme developers is nowhere to be found in the docs and not in the wiki-page either. I found it here after spending countless hours of experimenting with both kinds of syntax highlighting:

Strings and string placeholders might not be the typical use case for semantic highlighting . I'd recommend to leave that to the TextMate grammar. 👉The semantic highlighting is something to go on top of syntax highlighting👈 and typically focuses on symbols where a full AST and resolution is needed to evaluate the type of a symbol. E.g. only when resolving an identifier in the file or project it's possible to know if its member, parameter, function and so on.

As can be found HERE

repeat:

"The semantic highlighting is something to go on top of syntax highlighting"

That's actually nearly the same sentence that is somewhere high in the semantic highlight guide, but only in the given context of the cited comments it became clear to me that it is recommended to use semantic highlighting only where classic TextMate scoped don't work. Despite semantic is far superior in it's understanding of the syntax.

---

The property semanticHighlighting defines whether the theme is ready for highlighting using semantic tokens. It is false by default, but we encourage all themes to enable it. [*]

Don't forget however:

Semantic highlighting wins against syntax highlighting as the semantic token provider has a better understanding of the source than the regex based TextMate grammar. [*]

And that's cool because:

Semantic highlighting enriches the syntax coloring based on symbol information from a language service that has the full understanding of the project. Based on this understanding each identifier gets colored & styled with the color of the symbol it resolves to. A constant variable name is rendered as constant throughout the file, not just in its declaration. Same for parameter names, property names, class names and so on. [*]

---

So after finding all those more or less undocumented Information I'm here:

• semantic highlighting is encouraged
• but only for things I cannot reach with TextMate Syntax
• semantic highlighting makes it impossible for TextMate scopes to be use, when built in semantics are rather broad *
• it does not help to have higher TextMate specify, semantic always wins *
• adding custom semantic TextMate scope mappings is possible - but (to what I've guessed, experimented and understand) not within a theme alone, one needs to write a language extension. (Am I right?)
• it is not possible to only augment an existing grammar you need to replace it.* (I could see myself adding something to the typescript grammar to differentiate private, protected and public modifier keywords, but meddling with all the grammar just for that - no way. I do hope for semantic tokens that will help, but I don't have a good vibe about it. *)

Those are my key-learnings after a lot of effort. I do hope that my findings might help some others on their journey, and I would really appreciate it if my findings would result in improvements in the docs.

My Random Collection of Issues, Questions etc.:

• the wiki-page [dated May 3, 2020] states that TypeScript and JavaScript are the only languages with built in support for semantic highlighting, is this still true? (At the end of the page there are several outdated links...)
• decorators as semantic tokens do not work in typescript even if enabled in the complier. Or am I doing something wrong?
• the semantic highlight guides says: "The foreground needs to follow a color format as described in Color formats. Transparency is not supported." That is not / no longer true. The linked color formats include transparency and it also works with semantic highlightning.
• the last example in Custom TextMate scope mappings does not work in a theme on it's own, one needs a fitting language extension, is that right? (I tried with the mapping of "keyword.export": ["keyword.control.export.ts"] - no dice).
• the theme-sample uses a plist (with extension .tmTheme) It's from 2017 - is that still a supported option (if so, is there a documentation?) or is this simply outdated?
• the default themes contain semanticTokenColors for example for 'newOperator' etc. it seems they do not work, or am I mistaken?
• when I tried to learn from the default themes: I find that the scope inspector indicates for the parameters that they are are resolved as semantic tokens with type parameter and modifier declaration os variable.parameter respectively. (the same as in the example on top of the guide.) the foreground is in both cases following variable.paramter and the color #9CDCFE is applied. I cannot find such a token in any of the theme files. The only tokens using that color are textMate tokens defined in tokenColors and nothing in any of the
  semanticTokenColors of that sort. I'm confused by this! ❓❓❓🤷‍♂️ It seems to me that after all my research and experimenting I'm still missing crucial parts of the concept. Or does that mean that there is some kind of automatic reremapping of the predefined TextMate scope Mappings???

If a theme has semantic highlighting enabled, but does not contain a rule for the given semantic token, these TextMate scopes are used to find a TextMate theming rule instead.

it could make sense to me however if that would be:

If a theme has semantic highlighting enabled, but does not contain a rule for the given semantic token, these TextMate scopes are used to find a TextMate semantic theming rule instead.

---

I apologize for taking up so much room with this, but I sincerely hope that my sharing of my experience and findings can help others and encourage the team behind that great project to improve the issues I encountered. After all, if it is so hard to implement all the great features you develop the hard and inspired work you do for the community wil less valuable than it could and should be.