apache / apache/pulsar-dotpulsar

AOT (Ahead-of-Time) Compilation Compatibility Gaps

Open
#288 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
C#
Stars
271
Forks
87
PR merge metrics
No merged PRs in 30d

Description

### Description

DotPulsar is not compatible with .NET Native AOT publishing. Attempting to use it in a project built with `true` produces multiple trim and dynamic-code warnings at build time, and the produced binary may crash at runtime when compression or schema features are exercised.

The root causes fall into three areas:

1. **Compression plugins** — all five compression backends (`BuiltinZlibCompression`, `Lz4Compression`, `SnappyCompression`, `ZstdCompression`, `ZstdSharpCompression`) discover and invoke the compression libraries entirely through `Assembly.Load()` + runtime reflection. These patterns are fundamentally incompatible with AOT and trimming.
2. **Avro schemas** — `AvroISpecificRecordSchema` and `AvroGenericRecordSchema` also use `Assembly.Load("Avro")`, `DefinedTypes`, `MakeGenericType()`, and `Activator.CreateInstance()` to bind to the Apache Avro library at runtime.
3. **JSON schema** — `JsonSchemaDefinitionGenerator` inspects types via `GetProperties()` / `GetCustomAttribute()` / `Enum.GetNames()` at runtime, and `JsonSchema` calls `JsonSerializer.Deserialize` / `SerializeToUtf8Bytes` without a source-generated `JsonSerializerContext`, which is trim-unsafe.

The library also declares no AOT-compatibility stance (`` is absent from the `.csproj`) and carries no `[RequiresDynamicCode]` / `[RequiresUnreferencedCode]` annotations to alert consumers.

### Reproduction Steps

1. Create a new .NET 8/9/10 console application.
2. Reference DotPulsar (latest, 5.3.1 at time of writing).
3. Add `true` to the project file.
4. Call any API that touches compression or a JSON/Avro schema (or simply add `using DotPulsar;`).
5. Publish: `dotnet publish -r linux-x64 -c Release`

Build output (verified with `dotnet publish -r win-x64 -c Release` against DotPulsar 5.3.1 / .NET 9 SDK) includes **39 warnings** — a representative sample:

```
AvroISpecificRecordSchema.cs(114): Trim analysis warning IL2026: ...Assembly.DefinedTypes.get which has 'RequiresUnreferencedCodeAttribute'...
AvroISpecificRecordSchema.cs(136): AOT analysis warning IL3050: ...System.Type.MakeGenericType(Type[]) which has 'RequiresDynamicCodeAttribute'...
AvroISpecificRecordSchema.cs(136): Trim analysis warning IL2055: Call to 'System.Type.MakeGenericType(Type[])' can not be statically analyzed...
BuiltinZlibCompression.cs(28): Trim analysis warning IL2026: ...Assembly.GetTypes() which has 'RequiresUnreferencedCodeAttribute'...
BuiltinZlibCompression.cs(35): AOT analysis warning IL3050: ...System.Type.GetEnumValues() which has 'RequiresDynamicCodeAttribute'...
Lz4Compression.cs(34): Trim analysis warning IL2026: ...Assembly.DefinedTypes.get which has 'RequiresUnreferencedCodeAttribute'...
SnappyCompression.cs(32): Trim analysis warning IL2026: ...Assembly.DefinedTypes.get which has 'RequiresUnreferencedCodeAttribute'...
ZstdCompression.cs(33): Trim analysis warning IL2026: ...Assembly.DefinedTypes.get which has 'RequiresUnreferencedCodeAttribute'...
ZstdSharpCompression.cs(33): Trim analysis warning IL2026: ...Assembly.DefinedTypes.get which has 'RequiresUnreferencedCodeAttribute'...
JsonSchemaDefinitionGenerator.cs(53): Trim analysis warning IL2070: ...'this' argument does not satisfy 'PublicProperties' in call to 'System.Type.GetProperties(BindingFlags)'...
```

The full list is in the **Other information** section below.

At runtime (depending on trimming aggressiveness), the compressor factory initialization will throw `FileNotFoundException` (trimmed assembly) or `MissingMethodException`.

### Expected behavior

The library should either:
- Be fully AOT-compatible for the core messaging path (at minimum) and annotate any optional features (Avro, dynamic compression loading) with `[RequiresDynamicCode]` / `[RequiresUnreferencedCode]` so the compiler can warn consumers; **or**
- Declare `false` in the `.csproj` and document the limitation.

Ideally the core path (connect, produce, consume with `byte[]` / primitive schemas) would be AOT-safe, with opt-in AOT-unsafe features clearly annotated.

### Actual behavior

All five compression loaders use `Assembly.Load()` and extensive reflection with no annotations. The Avro schema implementations do the same. The JSON schema implementation uses un-annotated `System.Text.Json` reflection APIs. The project carries zero `[RequiresDynamicCode]` / `[RequiresUnreferencedCode]` attributes and no `` marker.

### Regression?

Not a regression — AOT compatibility has not been declared or targeted in any prior release.

### Known Workarounds

- Build without `` and instead use `` with `true` (dynamic loading still works).
- Suppress individual warnings with `` / `` in the consuming project (fragile, requires intimate knowledge of the library internals).
- For the JSON schema specifically, pass a pre-built `jsonSchemaDefinition` string to `JsonSchema(options, jsonSchemaDefinition)` and supply a source-generated `JsonSerializerContext` via `JsonSerializerOptions`.

### Configuration

- DotPulsar version: 5.3.1
- .NET SDK: 9.0 / 10.0 (both affected)
- OS: Windows 11, Linux x64 (both affected)
- Architecture: x64, ARM64

### Other information

### Detailed breakdown of AOT-incompatible code

#### 1. Compression — `Assembly.Load()` + runtime reflection

All five files under `src/DotPulsar/Internal/Compression/` follow the same pattern: load an optional assembly at runtime, enumerate its types by full name, and create delegates or instances via `Activator.CreateInstance` / `MethodInfo.CreateDelegate`. None of these patterns survive trimming.

| File | Key problematic APIs |
|------|----------------------|
| `BuiltinZlibCompression.cs:27` | `Assembly.Load("System.IO.Compression")`, `assembly.GetTypes()`, `Activator.CreateInstance(zlibStreamType!, ...)` (lines 60, 71) |
| `Lz4Compression.cs:32` | `Assembly.Load("K4os.Compression.LZ4")`, `assembly.DefinedTypes.ToArray()`, `lz4Codec.GetMethods(BindingFlags.Public \| BindingFlags.Static)`, `method.CreateDelegate(...)` |
| `SnappyCompression.cs:30` | `Assembly.Load("IronSnappy")`, `snappy.GetMethods(BindingFlags.Public \| BindingFlags.Static)`, `method.CreateDelegate(...)` |
| `ZstdCompression.cs:31` | `Assembly.Load("ZstdNet")`, `GetMethods(BindingFlags.Public \| BindingFlags.Instance)`, `Activator.CreateInstance(...)`, `wrapMethod.CreateDelegate(...)` |
| `ZstdSharpCompression.cs:31` | `Assembly.Load("ZstdSharp")`, same pattern as above |

**Suggested fix:** Take compile-time dependencies on the optional compression packages instead of loading them dynamically. Gate each implementation behind a `#if` / multi-targeting or expose a registration API (`builder.UseSnappy()`) that accepts a concrete `ICompressorFactory` / `IDecompressorFactory` supplied by the consumer. This is the pattern used by libraries like `Microsoft.Extensions.Caching.StackExchangeRedis`.

#### 2. Avro schemas — `Assembly.Load()` + `MakeGenericType` + `Activator.CreateInstance`

`AvroISpecificRecordSchema` (lines 54, 57, 59, 71–72, 113–114, 119–120, 136, 149, 156, 159) and `AvroGenericRecordSchema` use an identical late-binding strategy against the `Apache.Avro` NuGet package.

Notable calls:
- `Assembly.Load("Avro")` — the assembly is trimmed away unless explicitly preserved
- `assembly.DefinedTypes.ToArray()` — all types must survive trimming for this to work
- `type.MakeGenericType(typeof(T))` — `[RequiresDynamicCode]`
- `Activator.CreateInstance(_avroWriterTypeInfo, _avroSchema)` — `[RequiresUnreferencedCode]`
- `MethodInfo.Invoke(...)` in `Decode` / `Encode` hot paths — defeats inlining and is trim-unsafe

**Suggested fix:** Take a direct compile-time reference to `Apache.Avro` and use the concrete `SpecificDatumWriter` / `SpecificDatumReader` types. If the dependency must remain optional, annotate both schema classes with `[RequiresDynamicCode("Avro schema binding uses runtime reflection.")]` and `[RequiresUnreferencedCode(...)]`.

#### 3. `JsonSchemaDefinitionGenerator` — runtime type inspection

`src/DotPulsar/Schemas/JsonSchemaDefinitionGenerator.cs` calls `GetProperties(BindingFlags.Public | BindingFlags.Instance)` (line 53), `GetCustomAttribute()` (line 58), `GetCustomAttribute()` (line 81), `Enum.GetNames(type)` (line 135), `type.GetElementType()` (line 154), `type.GetGenericTypeDefinition()` (line 161), and `type.GetGenericArguments()` (lines 170, 178) — all trim-unsafe when the concrete `T` is not statically known.

**Suggested fix:** Replace with a `System.Text.Json` source-generated schema approach (available in .NET 9 via `JsonSchemaExporter` in `System.Text.Json`) or accept an explicit schema string and deprecate the reflection-based generator. Short-term: annotate `Generate(Type, ...)` with `[RequiresUnreferencedCode]`.

#### 4. `JsonSchema` — trim-unsafe `JsonSerializer` calls

`src/DotPulsar/Schemas/JsonSchema.cs` calls `JsonSerializer.Deserialize(array, _options)` (line 59) and `JsonSerializer.SerializeToUtf8Bytes(message, _options)` (line 70). Both overloads without a `JsonTypeInfo` parameter are decorated with `[RequiresUnreferencedCode]` in the BCL.

**Suggested fix:** Add an overload that accepts `JsonTypeInfo` (or a `JsonSerializerContext`) and route serialization through it. Mark the existing reflection-based constructors with `[RequiresUnreferencedCode]`.

#### 5. No `` declaration

The absence of `true` in `DotPulsar.csproj` means the .NET SDK trim analyzer is **not** run during normal builds of the library itself. Adding `true` (even as a goal/aspiration) would surface all of the above as build-time errors in the library's own CI, making it impossible to accidentally re-introduce these patterns.

---

### Full verified warning list

Obtained by running `dotnet publish -r win-x64 -c Release` against a minimal consumer project referencing DotPulsar 5.3.1 source (the native link step fails because MSVC is not on `PATH`, but all ILLink/AOT analyzer warnings are emitted before that step).

**39 warnings across 8 source files:**

| # | File | Line | Warning | Description |
|---|------|------|---------|-------------|
| 1 | `Schemas/AvroISpecificRecordSchema.cs` | 54 | IL2080 | `GetInterfaces()` — field `_typeT` not annotated with `DynamicallyAccessedMemberTypes.Interfaces` |
| 2 | `Schemas/AvroISpecificRecordSchema.cs` | 57 | IL2080 | `GetField()` — field `_typeT` not annotated with `PublicFields` |
| 3 | `Schemas/AvroISpecificRecordSchema.cs` | 71 | IL2075 | `GetProperty()` — return of `GetType()` not annotated with `PublicProperties` |
| 4 | `Schemas/AvroISpecificRecordSchema.cs` | 72 | IL2075 | `GetMethod()` — return of `GetType()` not annotated with `PublicMethods` |
| 5 | `Schemas/AvroISpecificRecordSchema.cs` | 114 | IL2026 | `Assembly.DefinedTypes` has `[RequiresUnreferencedCode]` — types may be removed |
| 6 | `Schemas/AvroISpecificRecordSchema.cs` | 119 | IL2065 | `GetMethods()` — `this` value not statically determinable |
| 7 | `Schemas/AvroISpecificRecordSchema.cs` | 120 | IL2065 | `GetMethods()` — same as above (second call) |
| 8 | `Schemas/AvroISpecificRecordSchema.cs` | 136 | IL3050 | `MakeGenericType()` has `[RequiresDynamicCode]` — native code for instantiation may be absent |
| 9 | `Schemas/AvroISpecificRecordSchema.cs` | 136 | IL2055 | `MakeGenericType()` — call cannot be statically analyzed |
| 10 | `Schemas/AvroISpecificRecordSchema.cs` | 149 | IL3050 | `MakeGenericType()` — same (reader type) |
| 11 | `Schemas/AvroISpecificRecordSchema.cs` | 149 | IL2055 | `MakeGenericType()` — same (reader type) |
| 12 | `Schemas/AvroISpecificRecordSchema.cs` | 156 | IL2077 | `Activator.CreateInstance()` — field `_avroWriterTypeInfo` not annotated with `PublicConstructors` |
| 13 | `Schemas/AvroISpecificRecordSchema.cs` | 159 | IL2077 | `Activator.CreateInstance()` — field `_avroReaderTypeInfo` not annotated with `PublicConstructors` |
| 14 | `Internal/Compression/BuiltinZlibCompression.cs` | 28 | IL2026 | `Assembly.GetTypes()` has `[RequiresUnreferencedCode]` |
| 15 | `Internal/Compression/BuiltinZlibCompression.cs` | 35 | IL3050 | `Type.GetEnumValues()` has `[RequiresDynamicCode]` |
| 16 | `Internal/Compression/BuiltinZlibCompression.cs` | 37 | IL3050 | `Type.GetEnumValues()` — same (second enum type) |
| 17 | `Internal/Compression/BuiltinZlibCompression.cs` | 60 | IL2067 | `Activator.CreateInstance()` — parameter `zlibStreamType` not annotated with `PublicConstructors` |
| 18 | `Internal/Compression/BuiltinZlibCompression.cs` | 71 | IL2067 | `Activator.CreateInstance()` — same (decompressor lambda) |
| 19 | `Internal/Compression/Lz4Compression.cs` | 34 | IL2026 | `Assembly.DefinedTypes` has `[RequiresUnreferencedCode]` |
| 20 | `Internal/Compression/Lz4Compression.cs` | 39 | IL2075 | `GetMethods()` — return of `FindLZ4Codec()` not annotated with `PublicMethods` |
| 21 | `Internal/Compression/SnappyCompression.cs` | 32 | IL2026 | `Assembly.DefinedTypes` has `[RequiresUnreferencedCode]` |
| 22 | `Internal/Compression/SnappyCompression.cs` | 36 | IL2075 | `GetMethods()` — return of `FindSnappy()` not annotated with `PublicMethods` |
| 23 | `Internal/Compression/ZstdCompression.cs` | 33 | IL2026 | `Assembly.DefinedTypes` has `[RequiresUnreferencedCode]` |
| 24 | `Internal/Compression/ZstdCompression.cs` | 36 | IL2075 | `GetMethods()` — return of `Find()` not annotated with `PublicMethods` |
| 25 | `Internal/Compression/ZstdCompression.cs` | 40 | IL2075 | `GetMethods()` — return of `Find()` not annotated with `PublicMethods` (second call) |
| 26 | `Internal/Compression/ZstdCompression.cs` | 45 | IL2072 | `Activator.CreateInstance()` — return of `Find()` not annotated with `PublicParameterlessConstructor` |
| 27 | `Internal/Compression/ZstdCompression.cs` | 55 | IL2072 | `Activator.CreateInstance()` — same (decompressor lambda) |
| 28 | `Internal/Compression/ZstdCompression.cs` | 83 | IL2075 | `TypeInfo.ImplementedInterfaces` — enumerator current not annotated with `Interfaces` |
| 29 | `Internal/Compression/ZstdCompression.cs` | 83 | IL2075 | `Type.GetConstructor()` — enumerator current not annotated with `PublicParameterlessConstructor` |
| 30 | `Internal/Compression/ZstdSharpCompression.cs` | 33 | IL2026 | `Assembly.DefinedTypes` has `[RequiresUnreferencedCode]` |
| 31 | `Internal/Compression/ZstdSharpCompression.cs` | 36 | IL2075 | `GetMethods()` — return of `FindDecompressor()` not annotated with `PublicMethods` |
| 32 | `Internal/Compression/ZstdSharpCompression.cs` | 40 | IL2075 | `GetMethods()` — return of `FindCompressor()` not annotated with `PublicMethods` |
| 33 | `Internal/Compression/ZstdSharpCompression.cs` | 45 | IL2072 | `Activator.CreateInstance()` — return of `FindCompressor()` not annotated with `PublicConstructors` |
| 34 | `Internal/Compression/ZstdSharpCompression.cs` | 55 | IL2072 | `Activator.CreateInstance()` — return of `FindDecompressor()` not annotated with `PublicParameterlessConstructor` |
| 35 | `Internal/Compression/ZstdSharpCompression.cs` | 83 | IL2075 | `TypeInfo.ImplementedInterfaces` — enumerator current not annotated with `Interfaces` (FindDecompressor) |
| 36 | `Internal/Compression/ZstdSharpCompression.cs` | 83 | IL2075 | `Type.GetConstructor()` — enumerator current not annotated with `PublicParameterlessConstructor` (FindDecompressor) |
| 37 | `Internal/Compression/ZstdSharpCompression.cs` | 103 | IL2075 | `TypeInfo.ImplementedInterfaces` — enumerator current not annotated with `Interfaces` (FindCompressor) |
| 38 | `Internal/Compression/ZstdSharpCompression.cs` | 103 | IL2075 | `Type.GetConstructor()` — enumerator current not annotated with `PublicConstructors` (FindCompressor) |
| 39 | `Schemas/JsonSchemaDefinitionGenerator.cs` | 53 | IL2070 | `GetProperties(BindingFlags)` — parameter `type` not annotated with `PublicProperties` |

> **Note:** `AvroGenericRecordSchema` and the `JsonSchema.Decode` / `Encode` paths were not reachable from the probe's entry point, so their warnings do not appear in this run. The actual warning count in a real application touching those code paths will be higher.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the warnings with `dotnet publish -r win-x64 -c Release`, then inspect the five files under `src/DotPulsar/Internal/Compression/`, the Avro schema files, `JsonSchemaDefinitionGenerator.cs`, `JsonSchema.cs`, and `DotPulsar.csproj`. Compare the possible AOT compatibility approaches in the issue and define done as a documented compatibility stance, addressed or explicitly annotated warnings, and a working core messaging path under Native AOT.

Written by the indexing model from the issue text.

Assessment

Tech stack
csharp
Domain
backend, build-system
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.