Expand semantic highlighting support for CompletionItem.Documentation and Hover.Contents

Version 3.16 will add semantic highlighting to the protocol, which is a great addition that Omnisharp has already used to great effect. However, adding this to the text editor has then made all the places that use markdown for formatting today stick out, as they're not using real semantic classification and can often mess up highlighting, particularly because they're just snippets. A bare word might refer to a parameter or field or method name, for example, but as the markdown content of the documentation or hover has no extra info, it'll be highlighted as a local variable. I recently implemented some improvements for Omnisharp in the hover provider to use Roslyn's QuickInfo service, but the lack of expressiveness in markdown quickly became apparent. I think the issue is best demonstrated through a picture. Given this simple sample with some documentation comments:

namespace ClassLibrary1
{
    public class Class1
    {
        /// <summary>
        /// Gets an instance of <see cref="Class2"/>
        /// </summary>
        /// <returns>A created instance of <see cref="Class2"/></returns>
        public Class2 M(string param1, int param2)
        {
            return null;
        }
    }
    public class Class2
    {
    }
}

Here's what the hover looks like in VS over M:

There's several pieces of this popup that are simply unachievable through unaltered markdown:

1. The method codicon is displayed inline with the method signature, and the signature is highlighted as C# code. You can't do this with markdown: content inside a code block is interpreted literally, so $(symbol-method) would be interpreted as that literal text and not substituted with a codicon.
2. All the components of the signature are clickable links that go to the definition of that component. You can click on any highlighted reference to Class2 and it will go to definition on that member. This is also not possible for the same reason as # 1: the []() link syntax inside a code block would be interpreted literally.
3. The references to Class2 in the summary and returns section are semantically colored and clickable. There are some markdown extensions that support inline colorization of single-line code blocks, but this is a) not semantic, and b) not guaranteed to be supported by the editor. You can get links working for these locations as they're single-line blocks, but you can't get both today.

To solve these, I'd like to propose an extension to MarkupContent. This is a very, very, very rough proposal: I spend most of my time in compiler land and am certain to get parts of this wrong, so criticism and ideas are welcome. This extension has, to my mind, a few specific goals:

1. Allow for inline semantic rendering of code text.
2. Allow for rich linking inside the content, able to execute commands based on clicking. This can be LSP commands, such as go-to-definition, but it should be extensible enough to allow for other things. URLs, for example, should be resolvable, as it's perfectly reasonable to put a web link in documentation.

My initial proposal is to add a new kind to MarkupKind to indicate that the content is semantic markdown and provide a SemanticTokens stream that should be used for colorization. In the definition below, I've elided the existing parts of MarkupContent for brevity, but assume they're unchanged except where noted.

/**
 * Describes the content type that a client supports in various
 * result literals like `Hover`, `ParameterInfo` or `CompletionItem`.
 *
 * Please note that `MarkupKinds` must not start with a `$`. This kinds
 * are reserved for internal usage.
 */
export namespace MarkupKind {
    /* (unchanged) */
    /**
     * Semantic markdown is a markdown string, with an accompanying set of semantic tokens to
     * aid in editor colorization of the string. The first token delta is relative to the start
     * of the string, and includes punctuation such as code fences.
     */
     export const SemanticMarkdown: 'semanticMarkdown' = 'semanticMarkdown';
}
export type MarkupKind = 'plaintext' | 'markdown' | ['semanticMarkdown', SemanticTokens];

/**
 * ... (unchanged)
 *
 * *Please Note* that clients might sanitize the return markdown. A client could decide to
 * remove HTML from the markdown to avoid script execution.
 *
 * If the kind is `semanticMarkdown`, the tokens it provides are used when rendering the tokens in
 * this string. The first token delta is considered from the start of `value` and semantic tokens
 * are mapped using the `SemanticTokensLegend` provided by the language server during registration.
 *
 * TODO: More examples and documentation
 * 
 */
export interface MarkupContent {
    /* (unchanged) */
}

For the previous example of hovering over M, this is a rough idea of what you'd provide. I'm going to just use type or identifier names in the url locations, but presumably they'd be actual URIs that link to commands.

$(symbol-method) [`Class2`](Class2) [`Class1`](Class1)`.`[`M`](M)`(`[`string`](string)` `[`param1`](param1)`, `[`int`](int)` `[`param2`](param2)`)`

Gets an instance of [`Class2`](Class2)

Returns:

  A created instance of [`Class2`](Class2)

The accompanying token stream would look something like this. I'm going to make the stream uncompressed for the sake of everyone reading this, and I'm sure I'll get some offset wrong, but it conveys the general idea:

[
  // Line 0
  { line: 0, startChar: 20, length: 5, tokenType: ["class"], tokenModifiers: ["public"] },            // [Class2] Class1.M(string param1, int param2)
  { line: 0, startChar: 39, length: 5, tokenType: ["class"], tokenModifiers: ["public"] },            // Class2 [Class1].M(string param1, int param2)
  { line: 0, startChar: 60, length: 1, tokenType: ["method"], tokenModifiers: ["public"] },           // Class2 Class1.[M](string param1, int param2)
  { line: 0, startChar: 71, length: 5, tokenType: ["keyword", "type"], tokenModifiers: ["public"] },  // Class2 Class1.M([string] param1, int param2)
  { line: 0, startChar: 92, length: 6, tokenType: ["parameter"], tokenModifiers: [] },                // Class2 Class1.M(string [param1], int param2)
  { line: 0, startChar: 114, length: 5, tokenType: ["keyword", "type"], tokenModifiers: ["public"] }, // Class2 Class1.M(string param1, [int] param2)
  { line: 0, startChar: 138, length: 6, tokenType: ["parameter"], tokenModifiers: [] },               // Class2 Class1.M(string param1, int [param2])

  // Line 2
  { line: 2, startChar: 23, length: 5, tokenType: ["class"], tokenModifiers: ["public"] },            // Gets an instance of [Class2]

  // Line 7
  { line: 2, startChar: 27, length: 5, tokenType: ["class"], tokenModifiers: ["public"] },            //  A created instance of [Class2]
]

As I said, this is a very rough first draft, and there are a few open questions I have:

1. I've tried to reuse markdown as a basis for the proposal to avoid having to come up with yet another markup language for things like text formatting and linking, but is it trying to stretch markdown too far?
2. Should things that need to be highlighted need to be wrapped in backticks? Or should the presence of a semantic classification for the item do all that a backtick implies?
   a. If yes, backticks should still be specified, should the semantic classification span include the backticks?
3. What happens if you provide semantic tokens for things inside a triple-backtick code block? In particular, what if that code block already has a language provided?
4. It would be very nice if language servers could provide additional information along with the semantic token to aid in handling commands. For example, in order to semantically highlight the string the server already had to look up the symbol representation, and it would be good if the information for going to that definition or performing other actions was able to be included with the token. I'm unsure how to best achieve this, however: would we want to include it in the URI link?

Start with the MarkupKind and MarkupContent definitions in the Language Server Protocol specification, then review the proposed semanticMarkdown representation and its open design questions. Done means reaching agreement on the markup, semantic-token mapping, linking behavior, and related protocol documentation for CompletionItem.Documentation and Hover.Contents.