[API Proposal]: Sharing pipeline values between incremental generators
- Dominant language
- C#
- Stars
- 20.7k
- Forks
- 4.3k
- PR merge metrics
- PR metrics pending
Description
## Background and Motivation
Generator output is added to the compilation once, after every generator has run, and only for the final build. The compilation generators observe never contains it: what one generator emits is invisible as symbols to every other generator and to itself. That is by design. Adding trees to a compilation produces a new `Compilation` with a new symbol graph and all binding redone; parsing is cheap and incremental, the semantic layer is not. Making each generator's output visible to the rest would multiply that cost by the number of generators, in the IDE on every keystroke. The one exception, `RegisterPreCompilationSourceOutput`, adds output before the compilation is built and is therefore limited to generators that do not need the compilation at all.
So instead of making generated code visible and paying for compilations, generators can exchange what they already share: the input compilation is the same for all of them. A consumer publishes a contract describing what it can act on, a producer fills it with data taken from that compilation, such as type names, and the consumer resolves those names in its own compilation and proceeds exactly as it does for an attribute today.
Most requests to "run generator B after generator A" have this shape: A knows what work needs to be done, B knows how to do it.
- A generator discovers types that need JSON serialization and wants the System.Text.Json generator to produce the serializers. It cannot generate them itself, and it cannot make the STJ generator see a `[JsonSerializable]` context inside its own output: STJ discovers contexts with `ForAttributeWithMetadataName` over the input compilation.
- A generator declares request types and wants the ASP.NET validations generator to produce `IValidatableInfo` for them. That generator collects types from `[ValidatableType]` and from the minimal API endpoints it can see, and neither source reaches generated code. ASP.NET ships an analyzer for this case; its message reads "Source generators cannot inspect each other's output. Declare the type in a regular .cs file instead."
Neither case is specific to third-party generators. dotnet/aspnetcore#56021 (open, milestone .NET 12) asks for the `JsonSerializerContext` to be populated automatically with the types minimal API endpoints use, and names "source generators that discover all API surface types" as one way to get there. The Request Delegate Generator already computes every parameter and response type of every endpoint; with a channel to the STJ generator it would be the first producer. Outside Microsoft, FastEndpoints ships a separate CLI tool that writes serializer contexts to disk for users to check in, and its documentation gives the reason: "a limitation in .NET source generation which prevents incremental source generators from being chained."
In both cases the producer needs the compilation to know what to ask for, because it reacts to attributes on user types. That rules out `RegisterPreCompilationSourceOutput` (#83089): pre-compilation moves source across generators before the compilation is built, this proposal moves data across generators after it is built.
Ordering proposals (#57239, #81395) were not declined for lack of demand. Each of them introduces an additional `Compilation`, and binding a compilation is the most expensive thing the driver does. The feedback on #81395 is explicit: "Making new compilations is precisely what we must not do with new SG apis."
This proposal stays inside that constraint. One generator publishes a stream of values of a contract type; another consumes that stream as an ordinary `IncrementalValuesProvider`, in the standard phase, against the same compilation every other generator sees. No new phase, no new compilation. Ordering between generators follows from data dependencies, the same way it already does between nodes inside one generator. Generators that do not use the feature are not affected.
## Proposed API
```diff
namespace Microsoft.CodeAnalysis
{
public readonly partial struct IncrementalGeneratorInitializationContext
{
+ // Producer side. Values from `provider` are published as T.
+ public void RegisterExternalOutput(IncrementalValuesProvider provider)
+ where T : IEquatable;
+
+ // Consumer side. Everything published as T by any generator in the run.
+ public IncrementalValuesProvider ExternalInputsProvider()
+ where T : IEquatable;
}
}
```
The contract type is the address. The generator that owns a contract defines `T` and ships it as a separate contracts package, a plain `netstandard2.0` library that a producer references from its generator project and bundles next to its own analyzer assembly. The driver matches producers to consumers by the namespace-qualified name of `T`.
`T` is a plain data type: records of primitives, strings and collections of those. A contract evolves the way an attribute surface does: additive changes only, and `[Obsolete]` plus a replacement record when the shape has to change.
Semantics:
- Contracts form a graph from producers to consumers. It must be acyclic.
- An external input resolves to an empty stream when no generator in the run produces the contract, or when the input takes part in a cycle. Only the cycle is reported, as a driver diagnostic naming the generators involved. A missing producer is silent, because removing a package must not break the build of a project that consumes its contract.
- A producer may compute its values from the compilation and from syntax providers. A consumer observes the producer's values from the same run.
- A contract with several producers resolves to the concatenation of their streams, in a deterministic order.
- A request names a type in the input compilation: declared by the user or in a referenced assembly. Generated types are not visible to consumers, and a consumer that cannot resolve a requested name reports a diagnostic rather than skipping it.
- The consumer's provider participates in incremental caching like any other provider. If a producer's stream is unchanged between runs, nothing downstream of it on the consumer's side is re-executed.
Driver changes are confined to `GeneratorDriver`. Every generator observes the same compilation, and the final compilation is assembled once from all outputs, as today.
## Usage Examples
### Requesting serializers from the STJ generator
Contract, shipped as `System.Text.Json.SourceGeneration.Contracts`:
```cs
namespace System.Text.Json.SourceGeneration
{
// TypeName is a fully qualified metadata name. ContextName is the metadata name of the
// JsonSerializerContext that should own the generated serializer. Whether STJ creates a
// context that user code has not declared is STJ's call; this example assumes it does,
// so the producer can reference the context from its own output.
public sealed record SerializableTypeRequest(string TypeName, string ContextName);
}
```
Producer, an endpoint generator that knows which types cross the wire:
```cs
public void Initialize(IncrementalGeneratorInitializationContext context)
{
var endpoints = context.SyntaxProvider.ForAttributeWithMetadataName(
"FluentEndpoints.EndpointAttribute",
static (node, _) => node is ClassDeclarationSyntax,
static (ctx, _) => EndpointModel.Create(ctx));
var wireTypes = endpoints
.SelectMany(static (e, _) => e.WireTypes)
.Select(static (t, _) => new SerializableTypeRequest(t.MetadataName, "FluentEndpoints.Generated.JsonContext"));
context.RegisterExternalOutput(wireTypes);
context.RegisterSourceOutput(endpoints, EmitEndpoint);
}
```
Consumer, inside the STJ generator:
```cs
var requested = context
.ExternalInputsProvider()
.Combine(context.CompilationProvider)
.Select(static (pair, ct) => ResolveRequest(pair.Left, pair.Right, ct));
var fromAttributes = /* existing [JsonSerializable] pipeline */;
context.RegisterSourceOutput(fromAttributes.Collect().Combine(requested.Collect()), EmitContexts);
```
`EndpointModel` and `WireTypes` are equatable records. Editing the body of a handler leaves the published stream unchanged, and the STJ generator does not re-run its emit step.
### Requesting validation info from the ASP.NET validations generator
Same shape on the producer side, with a `ValidatableTypeRequest(string TypeName)` contract owned by `Microsoft.Extensions.Validation`. The validations generator already merges two streams of types, one from `[ValidatableType]` and one from endpoint parameters, with `Concat` before it walks the type graph and emits. An external input is a third stream in the same place:
```cs
var validatableTypesFromInputs = context
.ExternalInputsProvider()
.Combine(context.CompilationProvider)
.Select(static (pair, ct) => ResolveValidatableType(pair.Left.TypeName, pair.Right, ct))
.Where(static type => !type.IsDefault);
var allValidatableTypesProviders = validatableTypesFromEndpoints
.Concat(validatableTypesWithAttribute)
.Concat(validatableTypesFromInputs);
```
Everything after the `Concat`, including `Distinct` and the emit step, is unchanged. The request types are ordinary user-declared classes; only the code that names them as validatable lives in generated source. That is why the attribute route is closed and the data route is open.
## Alternative Designs
Explicit ordering (#57239). Every downstream generator would see a new `Compilation` on every run, and ordering across NuGet packages has no natural owner.
Two-phase generators (#81395). Declarations from every generator are collected into an enriched compilation that every implementation phase observes. It solves the "see other generators' types" case in general, at the cost of one additional compilation for every project with generators, which is why it was sent back for rework.
`RegisterPreCompilationSourceOutput`. Already approved, and the right tool when the producer does not need the compilation. It cannot express a producer that reacts to attributes on user code.
Library-level composition. Works when the upstream logic is small enough to ship as a library and the downstream author is willing to call it. Does not work for STJ or the validations generator.
## Risks
External inputs carry data, not source. A consumer that has to bind against code the producer emitted is not served by this proposal, and that is left out on purpose.
Producer and consumer do not share a runtime type: the analyzer assembly loader loads each generator's dependencies into that generator's own `AssemblyLoadContext`, so a contract assembly referenced from two packages is loaded twice. The driver bridges that boundary by serializing values into the consumer's `T`, which is why `T` is restricted to plain data. A producer that drifted from the consumer's contract is therefore caught by the driver at run time, not by the compiler.
The feature has value only once the owner of a downstream generator adds a consumer branch. The downstream owner is the one who knows what a request should look like, so this is the right place for the work, but it means nothing ships until STJ or the validations generator adopt it. The validations generator is the cheapest first consumer, one more stream into an existing `Concat`; STJ with RDG as producer is the one that closes dotnet/aspnetcore#56021.
No breaking changes: the two methods are additive, and a run without producers or consumers behaves exactly as today. The performance cost for generators that do not use the feature is the graph construction over an empty set of contracts.
Contributor guide
Research direction
Start with GeneratorDriver and the proposed IncrementalGeneratorInitializationContext methods, then trace the producer and consumer examples for external inputs. Done means the design accounts for contract matching, dependency cycles, deterministic multiple producers, diagnostics, and incremental behavior without creating additional compilations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- csharp
- Domain
- compilers
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100