[API Proposal]: Custom dictionary support for Brotli

Start with draft implementation PR #118783, related issue #112656, and Brotli's shared_dictionary.h API linked in the proposal. Clarify the factory naming, attachment semantics, and SafeHandle lifetime/refcounting design; done means an agreed API and implementation that preserves dictionary lifetime for encoder, decoder, and stream use.

Background and motivation

Related to #112656
Draft implementation PR #118783

Brotli is a flexible compression algorithm which allows specifying custom dictionary to achieve better compression performance on particular types of data. One of the possible uses for this is https://datatracker.ietf.org/doc/draft-ietf-httpbis-compression-dictionary/.

Technical background

When attaching a dictionary to encoder/decoder, brotli library does not copy the provided dictionary data, so our implementation will have to ensure that

1. the dictionary data is kept alive and (preferably) unmodified
2. the dictionary data is pinned and is not moved by GC

For the above reason, this proposal creates a new class BrotliDictionary which user can construct, and then reuse across multiple encoder/decoder/streams.

When constructing a dictionary, the native API accepts an enum specifying the dictionary type

https://github.com/dotnet/runtime/blob/5fe72c12cd557197824c2485748c9e9a4af36b46/src/native/external/brotli/c/include/brotli/shared_dictionary.h#L35-L43

In theory, more than one dictionary formats may be supported in the future, so creating BrotliDictionary instances via factory method is preferred.

The way how a dictionary is attached to an encoder or decoder is assymetrical:

Encoder side accepts a separately constructed BrotliEncoderPreparedDictionary object.

BROTLI_ENC_API BrotliEncoderPreparedDictionary*
BrotliEncoderPrepareDictionary(BrotliSharedDictionaryType type,
    size_t data_size, const uint8_t data[BROTLI_ARRAY_PARAM(data_size)],
    int quality,
    brotli_alloc_func alloc_func, brotli_free_func free_func, void* opaque);

BROTLI_ENC_API BROTLI_BOOL BrotliEncoderAttachPreparedDictionary(
    BrotliEncoderState* state,
    const BrotliEncoderPreparedDictionary* dictionary);

Decoder side accepts a byte array.

BROTLI_DEC_API BROTLI_BOOL BrotliDecoderAttachDictionary(
    BrotliDecoderState* state, BrotliSharedDictionaryType type,
    size_t data_size, const uint8_t data[BROTLI_ARRAY_PARAM(data_size)]);

Both sides require the data array to be kept alive.

For simplicity, this proposal seeks to add the same signature for either side.

API Proposal

+    public sealed class BrotliDictionary : System.IDisposable
+    {
+        internal BrotliDictionary() { }

          // Alternative name: CreateFromRawBytes or just CreateFromBytes
          // ZstandardDictionary uses Create
+        public static System.IO.Compression.BrotliDictionary Create(System.ReadOnlySpan<byte> buffer, int quality) { throw null; }

          // Uses default quality
+        public static System.IO.Compression.BrotliDictionary Create(System.ReadOnlySpan<byte> buffer) { throw null; }

+        public void Dispose() { }
+    }

     public struct BrotliDecoder : System.IDisposable
     {
+        public void AttachDictionary(System.IO.Compression.BrotliDictionary dictionary) { }
     }

     public struct BrotliEncoder : System.IDisposable
     {
+        public void AttachDictionary(System.IO.Compression.BrotliDictionary dictionary) { }
     }

     public class BrotliStream
     {
+        public void AttachDictionary(System.IO.Compression.BrotliDictionary dictionary) { }
     }

API Usage

BrotliDictionary dictionary = BrotliDictionary.CreateFromBuffer(RawDictionaryData);

BrotliStream stream = new BrotliStream(....);
stream.AttachDictionary(dictionary);

// use stream as usual

Alternative Designs

ctor parameter instead of AttachDictionary method

Accepting dictionaries in Encoder/Decoder ctor would also be possible, but then we need to introduce a few more constructor overloads. Also, keep in mind that it is possible to attach multiple dictionaries in Brotli (the effect seems to be similar to concatenating the two dictionaries), so additional overloads accepting a dictionary collection might be needed.

+    public sealed class BrotliDictionary : System.IDisposable
+    {
          // proposal left unchanged
+    }

    public sealed class BrotliCompressionOptions
    {
        // If set, supersedes Quality
+       public BrotliDictionary? Dictionary { get; set; }
    }

     public struct BrotliDecoder : System.IDisposable
     {
+        public BrotliDecoder(System.IO.Compression.BrotliDictionary dictionary) { }
     }

     public struct BrotliEncoder : System.IDisposable
     {
+        public BrotliEncoder(System.IO.Compression.BrotliDictionary dictionary) { }
     }

     public class BrotliStream
     {
+        public BrotliStream(System.IO.Stream stream, System.IO.Compression.CompressionMode mode, BrotliDictionary dictionary, bool leaveOpen = false) { }
     }

An approach requiring fewer allocations is possible for decoders, as the native API only requires a reference to an array (that must be kept alive/pinned for the lifetime of the Decoder). But the design is more complicated and error prone, and if we assume that dictionaries are going to be long-lived and reused, does not offer enough savings.

Risks

Lifetime management of the BrotliDictionary (more specificly, the native memory held by the safe handle within) can be tricky. since brotli does not do any internal refcounting to ensure the dictionary will be alive long enough, we need to perform the refcounting in Managed code on the relevant SafeHandle types.