Source Generators and Analyzers extensibility

### **Motivation**

I'm exploring a scenario where the initialization context of a source generator (SGen) can subscribe to a delegate from an SGen engine host, provided by a method within the `SourceProducerContext`. This would enable core source generator authors to:

- **Check the state of user-defined metadata**
- **Allow third-party or the same authors to generate code** based on the information shared among authors of custom data through the proposed provider.
- **Integrate or extend existing generated code** by creating an execution mechanism, (could evolving into a tree-execution-queue).

However, this approach comes with performance concerns, such as potential deadlocks, especially when relying on extension SGen authors’ implementations. To mitigate these issues, one solution could involve propagating the cancel token and execute 3rd party SGens with some predefined timeout.

#### Key-points

- Coordination of 3rd party SGens, though can they cause waiting subsequent dependent (also the reason of the cancellation token and timeout, something you the team can consider to be taken differently)
- Avoid implementation collisions (namings and duplicate implementations)
- Obviously, compilation contexts (as well as nodes and syntaxes) can't be shared or accessed by different authors. (only metadata, not generation context)
- Subsequent generators must generate their own code only based on the given metadata shared by core SGens authors and can be combined with other providers
- Shared metadata should be an intermediary assembly not being the same core SGens assembly (not a good idea I guess)

---

### **Proposed API**

To address the outlined scenario, the following API can be introduced:

```csharp
public sealed class AnalyzerExtensibilityProvider
{
static IncrementalValueProvider Publish(T exchangedData, CancellationToken cancelToken);
static IncrementalValueProvider> SubscribeTo(Func exchangedData);
}

public sealed class AnalyzerSubscription
{
public T State { get; }
// Report back to the publisher the current state changes
public void Return(CancellationToken token);
}
```
> ### Names are proposal. Maybe you can provide better ones according to their usage context
- **`Publish`**: Allows the core source generator to publish data to third-party consumers.
- **`SubscribeTo`**: Allows third-party authors to subscribe to the data published by the core source generator.
- **`AnalyzerSubscription`**: Represents the subscription, allowing subscribers to return the state changes to the publisher.

---

### **Usage Sample**

#### **Core Source Generator Author Example**

```csharp
var compilationProvider = incrementalGeneratorInitializationContext.CompilationProvider;

// Usage of AnalyzerExtensibilityProvider as a property
var extensibilityProvider = incrementalGeneratorInitializationContext.AnalyzerExtensibilityProvider;

var collectedServiceContainerSymbols = context.SyntaxProvider
.ForAttributeWithMetadataName(ServiceContainerAttributeFQName,
(node, a) => true,
(t, c) => (t.SemanticModel, Class: (INamedTypeSymbol)t.TargetSymbol))
.Collect();

incrementalGeneratorInitializationContext.RegisterSourceOutput(
compilationProvider
.Combine(extensibilityProvider)
.Combine(collectedServiceContainerSymbols),
(sourceProducer, ((compilation, interop), collectedServiceContainer), cancellationToken) => {

HashSet namesSet = new(NameUniquenessComparer); //ensure file name uniquness name

foreach (var (semanticModel, serviceContainerSymbol) in collectedServiceContainer)
{
if(ServiceContainerGenerator
.DiscoverServicesFor(
compilation,
semanticModel,
serviceContainerSymbol,
namesSet,
unresolvedDependencyHandler: (ServiceMetadata serviceDesc) => {
// ServiceMetadata type is hosted at some core-author assembly
// to exchange custom metadata info with other interested SGen writers
interop.Publish(serviceDesc, cancellationToken);
})
.TryBuild(out string fileName, out string code))
{
sourceProducer.AddSource(fileName, code);
};
}
});
```

#### **Third-Party Source Generator Author Example**

```csharp
var compilationProvider = incrementalGeneratorInitializationContext.CompilationProvider;

// Usage of AnalyzerExtensibilityProvider as a property
var emittedLoggerServices = incrementalGeneratorInitializationContext.AnalyzerExtensibilityProvider
.SubscribeTo((serviceMetadataSub, cancelToken) =>
{
if (IsLogger(serviceMetadataSub.State))
{
serviceMetadataSubscription.State.Resolved = true;
serviceMetadataSubscription.State.Name = serviceMetadataSubscription.;
// Reports changes made to the current shared state for this SGen
// and no more changes can be reported in order to avoid delay on core-sgen publisher of this metadata
serviceMetadataSubscription.Return(cancelToken);

return true;
}
return false;
})
.Collect();

incrementalGeneratorInitializationContext.RegisterPostInitializationOutput(CreateLoggerFactory);

incrementalGeneratorInitializationContext.RegisterSourceOutput(
compilationProvider.Combine(emittedLoggerServices),
(sourceProducer, info, cancellationToken) => {
var (compilation, emittedLoggerServicesData) = info;
// Proceed to generate third-party code, with the help of the current Compilation context
});
```

---

### **Risks**

- **Performance Concerns**: The proposed mechanism could lead to potential deadlocks, particularly when extension SGen authors' implementations are not well-coordinated.
- **Broadcasting Solution**: As a potential mitigation, broadcasting shared types might alleviate some of these risks, but this too introduces its own challenges in managing the broadcast and ensuring consistent state across participants.

@sharwell ... At your considerations

Start by reviewing the proposed AnalyzerExtensibilityProvider API and the incrementalGeneratorInitializationContext usage examples in this issue. Read the source generator and analyzer extensibility design around these entry points, then determine whether a supported coordination mechanism can be specified without sharing compilation contexts or introducing deadlocks. Done would require an agreed API design and implementation scope, not just revised names.

Written by the indexing model from the issue text.