Clarify whether semantic tokens can include line endings in tokens

It's currently not clear from the spec whether a single-line semantic token is allowed to include the line ending in tokens.

I've tested this in VS Code and it seems to work fine, but I think it should be explicit. I'm somewhat hoping that this is allowed, because my parser can produce multiline tokens that need to be split up (only for the purposes of the LSP semantic tokens), and having to "carve out" the line endings is an additional complication. It also removes the ability for a client to infer that the whole range is the same token - should it wish to - rather than two tokens.

For example, given the code:

/**
 * foo
 * foo
 */

I am currently producing comment tokens with the contents:

• "/**\n"
• " * foo\n"
• " * foo\n"
• " */"

If this seems reasonable, I propose (and can send a PR) to add something like this to the spec:

The client capability multilineTokenSupport indicates whether a client supports multiline tokens. If not, tokens lengths may include the newline characters at the end of a line but should not be longer.

Or

The client capability multilineTokenSupport indicates whether a client supports multiline tokens. If not, tokens are assumed to be truncated at the end of the line even if the server indicates the length is longer.

Start by reviewing the semantic tokens specification and the six-comment issue thread, focusing on whether single-line token lengths may include line endings and how multilineTokenSupport affects that rule. Done means the specification states an unambiguous rule covering the examples in the issue and the proposed client behavior.