OTLP env vars silently dropped from ConfigMap when a project is referenced by others (Kubernetes publisher)

# OTLP environment variables silently dropped from ConfigMap when a project resource is referenced by other resources (Kubernetes publisher)

## Summary

When using the Kubernetes publisher (`Aspire.Hosting.Kubernetes`) in **publish mode**, a project resource that is **referenced by other resources** via `.WithReference(...)` (or any operation that attaches an `EndpointReference` to it) **silently loses its `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_PROTOCOL`, and `OTEL_SERVICE_NAME` environment variables** in the generated Helm chart's ConfigMap.

As a result, at runtime `MixingX.ServiceDefaults`' `ConfigureOpenTelemetry` sees no `OTEL_EXPORTER_OTLP_ENDPOINT` and skips `UseOtlpExporter()` — the Aspire Dashboard only receives telemetry from **unreferenced** services.

## Environment

- Aspire: `13.4.5` (also `13.4.5-preview.1.26316.12` for `Aspire.Hosting.Kubernetes`)
- AppHost: **polyglot / file-format** (`apphost.cs` with `#:sdk` / `#:package` directives)
- Publisher: `Aspire.Hosting.Kubernetes`
- OS: macOS (Apple Silicon), but the bug is publisher-side and OS-independent
- .NET: 10.0

## Reproduction

### Minimal AppHost

```csharp
#:sdk Aspire.AppHost.Sdk@13.4.5
#:package Aspire.Hosting.Kubernetes@13.4.5-preview.1.26316.12

var builder = DistributedApplication.CreateBuilder(args);

var k8s = builder.AddKubernetesEnvironment("k8s");
var registry = builder.AddContainerRegistry("registry", "localhost:5001");
k8s.WithContainerRegistry(registry);

// ServiceA — declared FIRST, will reference ServiceB (declared later) = "reverse reference"
var serviceA = builder.AddProject("service-a");

// ServiceB — declared SECOND, will be referenced by ServiceA
// (In the real project: sso declared first, gateway declared later,
// `sso.WithReference(gateway)` creates a reverse reference.)
var serviceB = builder.AddProject("service-b");

// 👇 THE BUG TRIGGER: serviceA references serviceB
serviceA.WithReference(serviceB);

builder.Build().Run();
```

### Steps

1. `aspire publish --output ./out --non-interactive`
2. Inspect `./out/templates/service-b/config.yaml`

### Expected

`service-b/config.yaml` should contain:
```yaml
OTEL_EXPORTER_OTLP_ENDPOINT: "{{ .Values.config.service_b.OTEL_EXPORTER_OTLP_ENDPOINT }}"
OTEL_EXPORTER_OTLP_PROTOCOL: "{{ .Values.config.service_b.OTEL_EXPORTER_OTLP_PROTOCOL }}"
OTEL_SERVICE_NAME: "{{ .Values.config.service_b.OTEL_SERVICE_NAME }}"
```

### Actual

Those three keys are **missing** from `service-b/config.yaml`. Meanwhile `service-a/config.yaml` (the referencer) and any *unreferenced* project both have them correctly.

## Real-world evidence (from MixingX.Hub, 7-service polyglot AppHost)

After `aspire publish`, the OTLP env presence per service's ConfigMap:

| Service | Referenced by others? | `OTEL_EXPORTER_OTLP_ENDPOINT` in ConfigMap |
|---|---|---|
| sso | no | ✅ present |
| opcua-peripheral | no | ✅ present |
| opcua-trimming | no | ✅ present |
| **gateway** | yes (by sso + agent via `WithReference`) | ❌ **missing** |
| **agent-0** | yes (by edge via `WithReference`) | ❌ **missing** |
| **edge-0** | yes (by agent via `WithReference`) | ❌ **missing** |
| **audit** | yes (by gateway via `WithReference`) | ❌ **missing** |

The correlation is exact: **every service referenced via `WithReference` loses OTLP; every unreferenced service keeps it.**

## Bisection experiments (what I ruled out)

I ran 5 publish experiments to isolate the trigger:

| # | How sso/agent reference gateway | gateway OTLP present? | Conclusion |
|---|---|---|---|
| baseline | `WithReference(gateway)` | ❌ | bug |
| 1 | `WithReference(gateway)` + gateway explicit `.WithOtlpExporter()` | ❌ | not an annotation-presence issue |
| 2 | remove both references | ✅ | **the reference itself is the trigger** |
| 3 | `WithReference(gateway.GetEndpoint("http"))` (endpoint ref) | ❌ | endpoint reference also triggers |
| 4 | `WithEnvironment(k, gateway.GetEndpoint("http"))` | ❌ | any `EndpointReference` triggers |
| 5 | `WithEnvironment(k, "http://gateway-service:8080")` (pure string) | ✅ | **pure string avoids the bug** |

**Conclusion:** the trigger is the target resource holding *any* `EndpointReference` originating from `gateway` (i.e. being in another resource's reference graph). Not `WithReference` per se.

## Root-cause investigation (decompiled from `Aspire.Hosting.Kubernetes.dll` + `Aspire.Hosting.dll`, v13.4.5)

### OTLP injection in publish mode comes from one place only

In publish mode, `OtlpConfigurationExtensions.RegisterOtlpEnvironment` is guarded by `if (!context.ExecutionContext.IsPublishMode)` — it does **not** inject OTLP env in publish mode. The publish-mode OTLP env comes entirely from:

```csharp
// Aspire.Hosting.Kubernetes.dll, KubernetesEnvironmentResource.PrepareDeploymentTargetsAsync (≈line 6417)
foreach (IResource r in GetComputeResources(appModel))
{
var env = GetComputeEnvironment(r);
if (env != null && env != this && env != OwningComputeEnvironment) continue;
if (DashboardEnabled)
{
var otlpEndpoint = Dashboard?.Resource.OtlpGrpcEndpoint;
if (otlpEndpoint != null) ConfigureOtlp(r, otlpEndpoint); // ← adds an EnvironmentCallbackAnnotation
}
await CreateKubernetesResourceAsync(r, ...);
}
```

```csharp
// ConfigureOtlp (≈line 6461)
private static void ConfigureOtlp(IResource resource, EndpointReference otlpEndpoint)
{
if (resource is IResourceWithEnvironment && resource.Annotations.OfType().Any())
{
resource.Annotations.Add(new EnvironmentCallbackAnnotation(context =>
{
context.EnvironmentVariables["OTEL_EXPORTER_OTLP_ENDPOINT"] = otlpEndpoint; // EndpointReference value
context.EnvironmentVariables["OTEL_EXPORTER_OTLP_PROTOCOL"] = "grpc"; // string value
context.EnvironmentVariables["OTEL_SERVICE_NAME"] = resource.Name; // string value
return Task.CompletedTask;
}));
}
}
```

### The callback IS registered and DOES evaluate correctly

I injected a diagnostic subscriber on `AfterPublishEvent` that re-evaluates each service's environment via both `GetEnvironmentVariableValuesAsync(Publish)` and a direct callback walk. **Both paths show gateway's env contains OTLP**:

```
[A-gateway] ExecutionConfigurationBuilder: env count=42, hasOTEL=True
[A-gateway] OTEL_EXPORTER_OTLP_ENDPOINT = {k8s-dashboard.bindings.otlp-grpc.url}
[A-gateway] OTEL_SERVICE_NAME = gateway
[A-opcua-peripheral] ExecutionConfigurationBuilder: env count=4, hasOTEL=True
[A-opcua-peripheral] OTEL_EXPORTER_OTLP_ENDPOINT = {k8s-dashboard.bindings.otlp-grpc.url}
```

The OtlpExporterAnnotation count is `1` on all services including gateway. So **the bug is NOT in callback registration or evaluation** — at the application-model level, gateway's environment correctly contains all three OTLP vars, identical to the working services.

### The bug is in ConfigMap serialization (KubernetesResource.ProcessEnvironmentAsync / ProcessValueAsync)

`KubernetesResource.ProcessEnvironmentAsync` (≈line 9412) iterates callbacks and writes results into the `EnvironmentVariables` dictionary that becomes the ConfigMap. For `OTEL_EXPORTER_OTLP_ENDPOINT`, the value is an `EndpointReference` (`otlpEndpoint` from `ConfigureOtlp`), which flows into `ProcessValueAsync` (≈line 9512):

```csharp
if (value is EndpointReference ep)
{
if (ComputeEnvironmentEndpointResolver.TryGetCrossEnvironmentEndpointExpression(ep, ..., out var expression))
{ value = expression; continue; }
var kr = (ep.Resource != this) ? await CreateKubernetesResourceAsync(ep.Resource, ...) : this;
EndpointMapping mapping = kr.EndpointMappings[ep.EndpointName]; // ← suspicious: KeyNotFound? swallowed?
return GetEndpointValue(mapping, EndpointProperty.Url);
}
```

**Hypothesis (not 100% confirmed without a debugger):** when `gateway` is referenced by multiple other resources, the cross-environment endpoint resolution / `CreateKubernetesResourceAsync(dashboard)` path mutates or conflicts with the dashboard resource's `EndpointMappings`, so resolving `otlpEndpoint` (`k8s-dashboard`'s `otlp-grpc` endpoint) fails or returns null during gateway's ConfigMap serialization. The three OTLP vars (whose values are `EndpointReference` + strings) are then silently absent from `gateway-config` ConfigMap, while the string-only `OTEL_DOTNET_EXPERIMENTAL_OTLP_RETRY` (from `WithProjectDefaults`) survives.

The surviving/missing split in the generated ConfigMap supports this:

| Env var | Value type | In gateway ConfigMap? |
|---|---|---|
| `OTEL_DOTNET_EXPERIMENTAL_OTLP_RETRY` | string `"in_memory"` | ✅ |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `EndpointReference` | ❌ |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | string `"grpc"` | ❌ (even though it's a string!) |
| `OTEL_SERVICE_NAME` | string `gateway.Name` | ❌ (even though it's a string!) |

The fact that the two **plain string** OTLP vars are *also* dropped (not just the `EndpointReference` one) suggests the whole `EnvironmentCallbackAnnotation` added by `ConfigureOtlp` is being lost/discarded during serialization when the resource is referenced, rather than just the one `EndpointReference`-typed value.

## Impact

- **Telemetry blind spot:** any service referenced by another service is invisible in the Dashboard in published K8s environments.
- **Silent failure:** no warning, no error in publish output. The vars just vanish.
- **Affects the most central services** in a typical architecture (gateways, API aggregators) — exactly the services you most want telemetry from.

## Workaround

In publish mode, avoid attaching `EndpointReference`s to the resource that needs OTLP. Inject the address as a pure string instead:

```csharp
if (builder.ExecutionContext.IsRunMode)
{
serviceA.WithReference(serviceB); // run mode: needed for service discovery env
}
else
{
// publish mode: pure string avoids the bug
serviceA.WithEnvironment("services__service-b__http__0", "http://service-b-service:8080");
}
```

## What would help confirm

Source-level answer to: in `KubernetesResource.ProcessEnvironmentAsync` → `ProcessValueAsync`, when the target resource is referenced by other resources (holds `EndpointReferenceAnnotation`s pointing into other resources), why does the `EnvironmentCallbackAnnotation` added by `ConfigureOtlp` (whose values are `EndpointReference` + 2 strings) fail to reach the ConfigMap `EnvironmentVariables` dictionary — while the same annotation on an unreferenced resource succeeds?

Thank you!