Analyzer API for consuming characters in embedded languages

## Background and Motivation

Reporting diagnostics in languages embedded in strings is highly nuanced. Computing the diagnostic's location within the source file is a large burden for analyzers. Various analyzers either solve this from scratch each time, or give up and provide a confusing and sub-par user experience by spreading the warning locations across the entire string.

Providing compiler support opens up the rich space of embedded language analysis. Embedded languages are common, such as XML, JSON, SQL, Regex, YAML, custom DSLs, and more. Removing this roadblock enables the community to ship more secure and correct code when working in these domains. When such community analyzers become significantly easier to build, and the community steps up to create or flesh them out, VS gains an advantage which other IDEs already provide in the embedded languages space.

Examples of embedded language analyzers which provide diagnostics and code fixes are , which analyzes SQL syntax inside a C# string, or , or a closed-source analyzer which does a similar thing while taking Dapper's syntax allowances into consideration. These analyzers error on invalid T-SQL syntax or provide warnings or suggestions for best practices such as requiring columns to be qualified.

When an embedded language analyzer reports a diagnostic, the source location for such diagnostics should be the relevant _subset_ of the string literal token. For example:

```cs
var widgets = await connection.QueryAsync("""
select Widgets.Id, Parts.Id, {|XYZ0001:Name|}
from dbo.Widgets
join dbo.Parts on Parts.WidgetId = Widgets.Id
""");
```

Where `{|XYZ0001:Name|}` signifies that the embedded text `Name` should be the location of a diagnostic XYZ0001.

In order to report a diagnostic, the analyzer must translate from _embedded_ string offsets within the unescaped string value (such as from `ILiteralOperation.ConstantValue`) to _source offsets_ within the C# file.

This translation currently requires the analyzer to understand and translate all escapes, as well as trimming raw string indentation _properly_. If `\u2014` appears in the C# string, the embedded language within the string does not receive a backslash, a lowercase U, and some hex digits; instead, the embedded language gains a single em dash character in this case. This is not solved once and done; C# recently added a new escape sequence, `\e`. C# also recently added raw string literals, and the analyzer is required to understand the intervening trimmed indentation.

- TSqlAnalyzer did not attempt to solve this problem, likely given its complexity without compiler support. The result is that it creates a diagnostic location which spans the entire string literal, making it hard to see where it's referring to in long multiline strings where the same name may appear many times.

- The DapperAOT analyzer has its own implementation for each language: ,

- The closed-source analyzer has its own implementation.

- Roslyn also has its own implementation. The complexity was high enough that the Roslyn implementation author had to sit down with the lexer implementation to work out all the cases to handle.

The compiler supports structured introspection of documentation comments; this type of syntax introspection is on similar footing.

## Proposed API

It could be a good idea to tie together the consumption of the unescaped characters with the source file span for each character. This would improve the API usability since you would no longer have to "just know" that you use GetOperation + ILiteralOperation.ConstantValue to get the unescaped string, and then get a service from some other place to map its internal offsets back to the file. Also, could there be a performance advantage in analyzing a SyntaxToken directly rather than going through IOperation to get it?

Roslyn has already solved cleanly with its internal [`IVirtualCharService`](https://sourceroslyn.io/#Microsoft.CodeAnalysis.Workspaces/IVirtualCharService.cs). `IVirtualCharService` powers Roslyn's own embedded language support for JSON, regular expressions, and others. The interface has stood the test of time. The existence of `Microsoft.CodeAnalysis.ExternalAccess.AspNetCore.EmbeddedLanguages.AspNetCoreCSharpVirtualCharService`, which is a copy of this API, also shows its usefulness.

The following API is a based on a simplified version of `IVirtualCharSequence` which enumerates UTF-16 chars (System.Char) instead of runes (System.Rune).

```diff
namespace Microsoft.CodeAnalysis
{
public readonly struct SyntaxToken : IEquatable
{
+ // Returns the constant string value of this token, with character positions correlated to source text
+ // positions. The starting and ending quotes are not included, escapes are decoded, and trimmed raw string
+ // indentation is removed. Examples of decoded escapes are \t and \u1234 in regular strings, "" in
+ // verbatim strings, and {{ or }} in interpolated string text content.
+ public EmbeddedString? EmbeddedText { get; }
}

+ public readonly struct EmbeddedString
+ {
+ // (Optional) The token that the EmbeddedString was produced from
+ public SyntaxToken Token { get; }
+
+ public int Length { get; }
+ public EmbeddedChar this[int index] { get; }
+
+ public EmbeddedString Slice(int start, int length);
+
+ // 'position' is based on the original SourceText that Token came from. Finds the EmbeddedChar in this string
+ // that contains the position (not just intersects it).
+ public EmbeddedChar? Find(int position);
+
+ // [Note: Produces the same string as ILiteralOperation.ConstantValue for the same syntax]
+ public override string ToString();
+
+ // (Could be omitted)
+ public Enumerator GetEnumerator();
+
+ public struct Enumerator : IEnumerator
+ {
+ public bool MoveNext();
+ public readonly EmbeddedChar Current { get; }
+ }
+ }
+
+ public readonly struct EmbeddedChar
+ {
+ public char Value { get; }
+
+ // The span within the original SourceText that produced this character.
+ // If the source text uses an escape for this character, the width will be greater than 1.
+ public TextSpan Span { get; }
+
+ // Allow `ch == 'A'` or `char.IsLetter(ch)` without having to include `.Value`
+ public static implicit operator char(EmbeddedChar ch);
+
+ // Returns `Value.ToString()`
+ public override string ToString();
+ }
+}
```

## Usage Examples

Roslyn's own internal usages of VirtualCharSequence and VirtualChar are informative, especially after @CyrusNajmabadi's recent PRs to refactor these APIs nearer to the shape of the proposed API above.

The following example is the closed-source analyzer rewritten to use the proposed API:

```cs
private void AnalyzeLiteralOperation(OperationAnalysisContext context)
{
if (!OperationFacts.HasStringSyntax(context.Operation, "T-SQL"))
return;

if (context.Operation.Syntax.GetFirstToken().EmbeddedText is not { } embeddedText)
return;

var syntaxContext = TSqlSyntaxContext.Parse(embeddedText.ToString());

foreach (var error in syntaxContext.ParseErrors)
{
TextSpan sourceErrorSpan;
if (error.Offset == embeddedText.Length)
{
sourceErrorSpan = new TextSpan(embeddedText[error.Offset - 1].Span.End, length: 0);
}
else
{
sourceErrorSpan = embeddedText[error.Offset].Span;

// The TSQL parser doesn't give lengths with the errors, so make the diagnostic as long as the single T-SQL
// token at the error position.
if (syntaxContext.GetTokenByOffset(error.Offset) is { } token)
{
sourceErrorSpan = TextSpan.FromBounds(
sourceErrorSpan.Start,
embeddedText[token.Offset + token.Text.Length - 1].Span.End);
}
}

context.ReportDiagnostic(Diagnostic.Create(
DiagnosticDescriptors.TSqlSyntaxParsingError,
Location.Create(context.Operation.Syntax.SyntaxTree, sourceErrorSpan),
[error.Number, error.Message]));
}

syntaxContext.Fragment?.Accept(new TSqlAnalyzerVisitor(
context,
fragment =>
{
var start = embeddedText[fragment.StartOffset].Span.Start;
var end = embeddedText[fragment.StartOffset + fragment.FragmentLength - 1].Span.End;
return Location.Create(context.Operation.Syntax.SyntaxTree, TextSpan.FromBounds(start, end));
}));
}
```

## Alternative Designs

We originally considered basing EmbeddedChar on runes instead of chars, but that was a step too far; some things naturally consume UTF-16 strings, and the most natural representation of what is in the source text (whether escaped or not) is UTF-16 characters. Use cases for runes can build on top of UTF-16 chars like they would in other scenarios.

## Risks

None identified specific to this proposal.