Can't lazely get a reference to a host application from a container when using the aspire container tunnel
- Dominant language
- C#
- Stars
- 6.3k
- Forks
- 991
- Avg merge
- 2d 15h
- Merged PRs (30d)
- 196
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues
### Describe the bug
If you don't have a reference to a host endpoint from a container in the app model before buildign the model, you can't later get one programmatically by calling `GetValueAsync` on the `EndpointReference`
### Expected Behavior
The endpoint should be lazily allocated within the aspire container tunnel, so that the url can be generated.
Alternatively, if the tunnel is immutable after the fact, the call should fail out right rather than hang indefinitely.
### Steps To Reproduce
```cs
var builder = DistributedApplication.CreateBuilder(args);
var exe = builder.AddExecutable("exe", "pwsh", ".")
.WithArgs("-NoLogo", "-NoProfile", "-Command", "while ($true) { Start-Sleep -Seconds 1 }")
.WithHttpEndpoint(name: "metrics");
var container = builder.AddContainer("nginx", "nginx");
// The final await call will will hang forever unless you either
// 1. Disable the container tunnel (see below) or
// builder.Configuration["ASPIRE_ENABLE_CONTAINER_TUNNEL"] = "false";
//
// 2. add a reference to the exe before building the model
// container.WithReference(exe.GetEndpoint("metrics"));
await builder.Build().StartAsync();
Console.WriteLine("Getting metrics url from view of container");
var url = await exe.GetEndpoint("metrics").Property(EndpointProperty.HostAndPort).GetValueAsync(new() { Caller = container.Resource }, default);
Console.WriteLine(url);
```
### Exceptions (if any)
_No response_
### Aspire doctor output
Aspire Environment Check
========================
Aspire
✅ Aspire CLI version 13.4.2 (channel: stable)
AppHost
✅ AppHost version 13.4.6 (..\dotnet\connector\aspire\AppHost\AppHost.csproj)
.NET SDK
✅ .NET 10.0.301 installed (x64)
Container Runtime
✅ Docker v29.6.1: running (auto-detected (default)) ← active
Environment
✅ HTTPS development certificate is trusted
Summary: 5 passed, 0 warnings, 0 failed
Aspire CLI Installations
========================
╭─────────────────────────────────────────┬────────────────────────────────────────┬─────────┬───────────┬─────────────╮
│ Path │ Version │ Channel │ Route │ PATH status │
├─────────────────────────────────────────┼────────────────────────────────────────┼─────────┼───────────┼─────────────┤
│ C:\Users\alexanderc\.aspire\bin\aspire. │ 13.4.2+d7d0b6759ce4b936c76bc4775814d27 │ stable │ (unknown) │ active │
│ exe (current) │ db560dd6d │ │ │ │
╰─────────────────────────────────────────┴────────────────────────────────────────┴─────────┴───────────┴─────────────╯
### Anything else?
A ~~bit~~ lot of context
I have a number of resources which expose their metrics via Prometheus and not native open telemetyr. I want to expose these resoruces in Aspire.
To do so, I have a process that uses an open telemetry collector instance that can import metrics from Prometheus, and export them through otel.
```yaml
receivers:
prometheus:
trim_metric_suffixes: true
config:
global:
scrape_interval: ${env:OTEL_METRIC_EXPORT_INTERVAL:-60000}ms
scrape_native_histograms: true
convert_classic_histograms_to_nhcb: true
scrape_protocols: [ PrometheusProto, OpenMetricsText1.0.0, OpenMetricsText0.0.1, PrometheusText0.0.4 ]
scrape_configs: ${file:${env:CONFIG_PATH}scrape-config.yaml}
exporters:
otlp_grpc/aspire:
endpoint: ${env:ASPIRE_ENDPOINT}
headers:
'x-otlp-api-key': ${env:ASPIRE_API_KEY:-dummy}
service:
pipelines:
metrics/prometheusToOtel:
receivers: [prometheus]
exporters: [otlp_grpc/aspire]
```
Where things get tricky is I also need to include the otel instance id when exporting metrics. This value isn't known until a resource starts, and even worse it changes every time you restart a resource. This means I can't build the configurations up front, and need to do so dynamically. So to do this, I end up (ab)using file-based service discovery - https://prometheus.io/docs/guides/file-sd/ for configuring scrape jobs.
For resources I want to convert Prometheus metrics to otel, I have an annotation I apply to the resource + extension method
```cs
public static IResourceBuilder WithPrometheusMetrics(this IResourceBuilder builder, EndpointReference endpoint, string? path)
where T : IResourceWithEndpoints
{
builder.ApplicationBuilder.TryAddPrometheusScraper();
return builder.WithAnnotation(new PrometheusScrapeAnnotation(endpoint) { Path = path });
}
```
On resource startup, the initial scrape config is built by scraping all resources with the annotation, and adding them to a list
```cs
.WithContainerFiles(configMount, async (ctx, ct) =>
{
var model = ctx.ServiceProvider.GetRequiredService();
return [
new ContainerFile{
Name = "scrape-config.yaml",
Contents = GetScrapeConfig()
}
];
string GetScrapeConfig()
{
var config = PrometheusConfigBuilder.BuildServiceDiscoveryJobs(model);
JsonSerializerOptions options = CreateJsonOptions();
return JsonSerializer.Serialize(config, options);
}
}
public static IReadOnlyList BuildServiceDiscoveryJobs(DistributedApplicationModel model)
{
List jobs = [.. model.Resources
.SelectMany(x => x.Annotations.OfType())
.Select(annotation => new Job
{
JobName = GetJobName(annotation),
Scheme = annotation.Endpoint.Scheme,
MetricsPath = annotation.Path,
FileSdConfigs =
[
new FileSdConfig
{
Files = [$"${{env:SERVICE_DISCOVERY_PATH}}/{GetJobName(annotation)}.json"],
RefreshInterval = "1s"
}
]
})];
if (jobs.Count == 0)
{
throw new InvalidOperationException("No resources are configured to be scraped");
}
return jobs;
}
```
resulting in a config file that looks like
```json
[
{
"job_name": "RESOURCE-1-metrics",
"scheme": "http",
"file_sd_configs": [
{
"files": [
"${env:SERVICE_DISCOVERY_PATH}/RESOURCE-1-metrics.json"
],
"refresh_interval": "1s"
}
]
},
{
"job_name": "RESOURCE-2-metrics",
"scheme": "http",
"file_sd_configs": [
{
"files": [
"${env:SERVICE_DISCOVERY_PATH}/RESOURCE-2-metrics.json"
],
"refresh_interval": "1s"
}
]
}
]
```
`SERVICE_DISCOVERY_PATH` is a directory that is BIND Mounted in so that I can modify the directory from the host. Now when instances start & stop, I dynamically add/remove the per service discovery file. I have to do this dynamically as I need to include the service id to properly link up metrics in the aspire dashboard, but this changes on every invocation.
```json
[
{
"targets": [
"RESOURCE-1.dev.internal:8000"
],
"labels": {
"job": "RESOURCE-1",
"instance": "{resource otel service id}",
"otel_scope_name": "metrics"
}
}
]
```
In which I'm generating targest with the following
```cs
public static async Task BuildServiceDiscoveryTarget(
PrometheusScrapeAnnotation annotation,
string? resourceId,
IResource caller,
CancellationToken cancellationToken)
{
var resourceName = annotation.Endpoint.Resource.Name;
string?[] targets = resourceId is null
? []
: [await annotation.Endpoint.Property(EndpointProperty.HostAndPort).GetValueAsync(new() { Caller = caller }, cancellationToken)];
var resourcePrefix = $"{resourceName}-";
var instance = resourceId?.StartsWith(resourcePrefix, StringComparison.Ordinal) == true
? resourceId[resourcePrefix.Length..]
: resourceId;
return new TargetConfig
{
Targets = targets,
Labels = {
["job"] = resourceName,
["instance"] = instance,
["otel_scope_name"] = annotation.Endpoint.EndpointName,
},
};
}
```
And this is where I hit problems - `await annotation.Endpoint.Property(EndpointProperty.HostAndPort).GetValueAsync(new() { Caller = caller }, cancellationToken)` hangs
Contributor guide
Assessment
This issue has not been assessed yet.