[API Proposal]: Add methods to `JsonElement` and `JsonProperty` to copy both raw and unescaped strings to `Span<byte>` and `Span<char>` buffers.

Start by reviewing the existing JsonMarshal APIs and the internal JsonReaderHelper capabilities used by Utf8JsonReader. Compare the proposed JsonElement and JsonProperty members with those paths, then determine the tests needed to validate escaped and raw copying, UTF8 and char destinations, buffer sizing, and invalid-value behavior.

Background and motivation

JsonMarshal has recently been added with APIs to get a raw UTF8 JsonElement value, and a raw UTF8 JsonProperty name.

In the rest of this section, to avoid repetition, I will use the word 'value' to mean either 'raw UTF8 value' or 'raw UTF8 property name', depending on context.

Building on this, in low allocation "raw value" processing, the three most common follow-up requirements (in order of usefulness) are:

1. Unescape the value if it contains any escape sequences into a stack-allocated or rented UTF8 byte buffer, before further processing. In the event that unescaping is not required, we would like to avoid the copy to the buffer.
2. Unescape the value if it contains any escape sequences, and transcode into a stack-allocated or rented char buffer before further processing.
3. Transcode the escaped value into a stack-allocated or rented char buffer before further processing.

Internally, JsonReaderHelper provides these capabilities to e.g. Utf8JsonReader but the libraries do not currently expose such functionality.

Considerations for the consumer

To unescape (and/or transcode) we need to know:

1. Is the value escaped?
2. How big a buffer is required for the unescaped/transcoded value?

Consumers will typically:

1. Get the information you require to perform the unescape and/or transcode operation
2. Decide whether you still want/need to perform the operation on that basis.

API Proposal

namespace System.Text.Json;

public readonly struct JsonElement
{

        /// <summary>
        ///    Indicates whether the value is escaped.
        /// </summary>
        /// <exception cref="ObjectDisposedException">
        ///   The parent <see cref="JsonDocument"/> has been disposed.
        /// </exception>
        /// <seealso cref="CopyString(Span{byte})"/>
        /// <seealso cref="CopyString(Span{char})"/>
        /// <seealso cref="CopyRawString(Span{byte})"/>
        /// <seealso cref="CopyRawString(Span{char})"/>
        public bool ValueIsEscaped { get; }

        /// <summary>
        ///   Copies the unescaped string value to the UTF8 string buffer.
        /// </summary>
        /// <param name="utf8Destination">The buffer into which to copy the string.</param>
        /// <returns>The number of bytes written to the output buffer, or 0 if the buffer was too small.</returns>
        /// <exception cref="InvalidOperationException">
        ///   This value's <see cref="ValueKind"/> is not <see cref="JsonValueKind.String"/>.
        /// </exception>
        /// <exception cref="ObjectDisposedException">
        ///   The parent <see cref="JsonDocument"/> has been disposed.
        /// </exception>
        public int CopyString(Span<byte> utf8Destination);
        
        /// <summary>
        ///   Copies the unescaped string value to the string buffer.
        /// </summary>
        /// <param name="destination">The buffer into which to copy the string.</param>
        /// <returns>The number of characters written to the output buffer, or 0 if the buffer was too small.</returns>
        /// <exception cref="InvalidOperationException">
        ///   This value's <see cref="ValueKind"/> is not <see cref="JsonValueKind.String"/>.
        /// </exception>
        /// <exception cref="ObjectDisposedException">
        ///   The parent <see cref="JsonDocument"/> has been disposed.
        /// </exception>
        public int CopyString(Span<char> destination);

        /// <summary>
        ///   Copies the raw string value to the UTF8 string buffer.
        /// </summary>
        /// <param name="utf8Destination">The buffer into which to copy the UTF8 string.</param>
        /// <returns>The number of bytes written to the output buffer, or 0 if the buffer was too small.</returns>
        /// <exception cref="InvalidOperationException">
        ///   This value's <see cref="ValueKind"/> is not <see cref="JsonValueKind.String"/>.
        /// </exception>
        /// <exception cref="ObjectDisposedException">
        ///   The parent <see cref="JsonDocument"/> has been disposed.
        /// </exception>
        /// <seealso cref="ToString"/>
        public int CopyRawString(Span<byte> utf8Destination)

        /// <summary>
        ///   Copies the raw string value to the string buffer.
        /// </summary>
        /// <param name="destination">The buffer into which to copy the string.</param>
        /// <returns>The number of characters written to the output buffer, or 0 if the buffer was too small.</returns>
        /// <exception cref="InvalidOperationException">
        ///   This value's <see cref="ValueKind"/> is not <see cref="JsonValueKind.String"/>.
        /// </exception>
        /// <exception cref="ObjectDisposedException">
        ///   The parent <see cref="JsonDocument"/> has been disposed.
        /// </exception>
        /// <seealso cref="ToString"/>
        public int CopyRawString(Span<char> destination)
}

public readonly struct JsonProperty
{
        /// <summary>
        ///   Indicates whether the name of this property is escaped.
        /// </summary>
        /// <seealso cref="CopyName(Span{byte})"/>
        /// <seealso cref="CopyName(Span{char})"/>
        /// <seealso cref="CopyRawName(Span{byte})"/>
        /// <seealso cref="CopyRawName(Span{char})"/>
        public bool NameIsEscaped { get; }

        /// <summary>
        ///   Copies the unescaped property name to the UTF8 string buffer.
        /// </summary>
        /// <param name="utf8Destination">The buffer into which to copy the string.</param>
        /// <returns>The number of bytes written to the output buffer, or 0 if the buffer was too small.</returns>
        /// <exception cref="InvalidOperationException">
        ///   This value's <see cref="Type"/> is not <see cref="JsonTokenType.PropertyName"/>.
        /// </exception>
        /// <exception cref="ObjectDisposedException">
        ///   The parent <see cref="JsonDocument"/> has been disposed.
        /// </exception>
        public int CopyName(Span<byte> utf8Destination);

        /// <summary>
        ///   Copies the unescaped property name to the string buffer.
        /// </summary>
        /// <param name="destination">The buffer into which to copy the string.</param>
        /// <returns>The number of characters written to the output buffer, or 0 if the buffer was too small.</returns>
        /// <exception cref="InvalidOperationException">
        ///   This value's <see cref="Type"/> is not <see cref="JsonTokenType.PropertyName"/>.
        /// </exception>
        /// <exception cref="ObjectDisposedException">
        ///   The parent <see cref="JsonDocument"/> has been disposed.
        /// </exception>
        public int CopyName(Span<char> destination);

        /// <summary>
        ///   Copies the raw property name to the string buffer.
        /// </summary>
        /// <param name="utf8Destination">The buffer into which to copy the UTF8 string.</param>
        /// <returns>The number of bytes written to the output buffer, or 0 if the buffer was too small.</returns>
        /// <exception cref="InvalidOperationException">
        ///   This value's <see cref="Type"/> is not <see cref="JsonTokenType.PropertyName"/>.
        /// </exception>
        /// <exception cref="ObjectDisposedException">
        ///   The parent <see cref="JsonDocument"/> has been disposed.
        /// </exception>
        public int CopyRawName(Span<byte> utf8Destination)

        /// <summary>
        ///   Copies the raw property name to the string buffer.
        /// </summary>
        /// <param name="destination">The buffer into which to copy the string.</param>
        /// <returns>The number of characters written to the output buffer, or 0 if the buffer was too small.</returns>
        /// <exception cref="InvalidOperationException">
        ///   This value's <see cref="Type"/> is not <see cref="JsonTokenType.PropertyName"/>.
        /// </exception>
        /// <exception cref="ObjectDisposedException">
        ///   The parent <see cref="JsonDocument"/> has been disposed.
        /// </exception>
        public int CopyRawName(Span<char> destination)
}

API Usage

The APIs support the common patterns of working with spans, making it easy to stack allocate, or rent/return value buffers.

We avoid throwing exceptions, except for the usual invalid cases.

Working with fully unescaped UTF8 JsonProperty names

JsonProperty property; // acquired by whatever means
const int StackAllocThreshold = 1024;

scoped ReadOnlySpan<byte> nameToProcess = JsonMarshal.GetRawUtf8PropertyName(property);
byte[]? unescapedArray = null;

if (property.NameIsEscaped)
{
    // This is the "slow path" where we have to unescape
    int bufSize = valueToProcess.Length;
    Debug.Assert(bufSize > 0); // we should never hit this
    Span<byte> utf8Unescaped =
        bufSize <= StackAllocThreshold
            ? stackalloc byte[bufSize]
            : (unescapedArray = ArrayPool<byte>.Shared.Rent(bufSize));

    written = property.CopyName(utf8Unescaped);
    Debug.Assert(written > 0); // We should never hit this

    nameToProcess = utf8Unescaped.Slice(0, written);
}

// Process the unescaped Name however we wish
nameToProcess.SequenceEqual("Boo!"u8);

// Return the rented buffer (if any)
if (unescapedArray is byte[] ua)
{
    ua.AsSpan(0, nameToProcess.Length).Clear();
    ArrayPool<byte>.Shared.Return(ua);
}

Working with fully unescaped UTF8 JsonElement values

JsonElement element; // obtained by whatever means
scoped ReadOnlySpan<byte> valueToProcess = JsonMarshal.GetRawUtf8Value(element);
byte[]? unescapedArray = null;

if (element.ValueIsEscaped)
{
    // This is the "slow path" where we have to unescape
    int bufSize = valueToProcess.Length;
    Debug.Assert(bufSize > 0); // we should never hit this
    Span<byte> utf8Unescaped =
        bufSize <= 1024
            ? stackalloc byte[bufSize]
            : (unescapedArray = ArrayPool<byte>.Shared.Rent(bufSize));

    written = element.CopyString(utf8Unescaped);
    Debug.Assert(written > 0); // we should never hit this

    valueToProcess = utf8Unescaped.Slice(0, written);
}

// Process the unescaped value however we wish
valueToProcess.SequenceEqual("Boo!"u8);

// Return the rented buffer (if any)
if (unescapedArray is byte[] ua)
{
    ua.AsSpan(0, valueToProcess.Length).Clear();
    ArrayPool<byte>.Shared.Return(ua);
}

Transcoding a raw JSON value, ensuring it is unescaped:

JsonElement element; // obtained by whatever means

bufSize = JsonMarshal.GetRawUtf8Value(element).Length;
System.Diagnostics.Debug.Assert(bufSize > 0); // should never hit this

char[]? transcodedArray = null;
Span<char> transcoded =
    bufSize <= 1024
        ? stackalloc char[bufSize]
        : (transcodedArray = ArrayPool<char>.Shared.Rent(bufSize));

written = element.CopyString(transcoded);
System.Diagnostics.Debug.Assert(written > 0); // should never hit this

ReadOnlySpan<char> transcodedValueToProcess = transcoded.Slice(0, written);

// Process the transcoded, unescaped value however we wish
transcodedValueToProcess.Equals("connectionId".AsSpan(), StringComparison.Ordinal);

if (transcodedArray is char[] ta)
{
    ta.AsSpan(0, transcodedValueToProcess.Length).Clear();
    ArrayPool<char>.Shared.Return(ta);
}

Transcoding a raw JSON value, but leaving it escaped.

JsonElement element; // obtained by whatever means

bufSize = JsonMarshal.GetRawUtf8Value(element).Length;
System.Diagnostics.Debug.Assert(bufSize > 0); // should never hit this

char[]? transcodedArray = null;
Span<char> transcoded =
    bufSize <= 1024
        ? stackalloc char[bufSize]
        : (transcodedArray = ArrayPool<char>.Shared.Rent(bufSize));

written = element.CopyRawString(transcoded);
System.Diagnostics.Debug.Assert(written > 0); // should never hit this

ReadOnlySpan<char> transcodedValueToProcess = transcoded.Slice(0, written);

// Process the transcoded, escaped value however we wish
transcodedValueToProcess.Equals("conne\\u0063tionId".AsSpan(), StringComparison.Ordinal);

if (transcodedArray is char[] ta)
{
    ta.AsSpan(0, transcodedValueToProcess.Length).Clear();
    ArrayPool<char>.Shared.Return(ta);
}

Alternative Designs

We previously discussed adding these methods to JsonMarshal, but as they conform to the backing-agnostic requirements of JsonProperty/JsonElement it seems appropriate to add them here. (See discussion below.)

It was suggested in comments that we could just use JsonMarshal.GetRawUtf8Value(...).Length to get a suitable buffer length for copying. This works well, especially for efficiency when processing UTF8 strings as you will require the raw UTF8 value anyway.

An alternative would be to provide GetBufferLength() and GetNameBufferLength() APIs on JsonElement/JsonProperty respectively. While you may not always use them (e.g. in the case where we are already retrieving the raw UTF8 value as part of our processing strategy) it would give a complete API on JsonElement/JsonProperty which might be more discoverable.

Risks

Changes to existing APIs

This would mean promoting both JsonElement.ValueIsEscaped and JsonProperty.NameIsEscaped from internal to public.

General risks

I do not believe there are any significant risks to this API, in terms of either performance, or breaking changes.

Implementation risks and constraints

As this is intended to be a high-performance, low-allocation API, it is worth considering the influence the API design might have on implementation options.

Existing JSON reader helper methods can be used to implement the public APIs trivially.