microsoft / microsoft/agent-framework
.NET: [Bug]: Hosted-agent runtime never activates OTLP export to a third-party destination, despite docs describing it as automatic
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.6k
- Forks
- 2.3k
- Avg merge
- 2d 45m
- Merged PRs (30d)
- 358
Description
Description
A .NET Foundry hosted agent (AgentHost.CreateBuilder + Microsoft.Agents.AI.Foundry.Hosting, zero manual OpenTelemetry instrumentation code, per the "zero-code telemetry" model this package advertises) never delivers OTLP data to a third-party destination via the documented OTEL_EXPORTER_OTLP_* environment variables, even though the exact same variables work correctly outside the hosted-agent runtime.
Per configure-hosted-agent-telemetry: "Hosted agents support the standard environment variables defined in the OpenTelemetry SDK configuration documentation... When OTEL_EXPORTER_OTLP_ENDPOINT is present, the runtime also exports over OTLP to that endpoint." This does not happen for .NET.
What happened:
- Deployed a hosted agent with these env vars set on the agent version (via
azure.yaml'senvmap, applied throughazd deploy):OTEL_EXPORTER_OTLP_ENDPOINT(generic) — tried both with and without this set, across 4 separate deployed agent versions.OTEL_EXPORTER_OTLP_PROTOCOL=http/protobufOTEL_EXPORTER_OTLP_TRACES_ENDPOINT/OTEL_EXPORTER_OTLP_METRICS_ENDPOINT/OTEL_EXPORTER_OTLP_LOGS_ENDPOINT(per-signal, full paths)OTEL_EXPORTER_OTLP_HEADERS/OTEL_EXPORTER_OTLP_TRACES_HEADERS(auth headers for the destination)
- A
GETon the deployed agent version (.../agents/{name}/versions/{n}?api-version=v1) confirmed every one of these variables was registered on the container exactly as configured, byte-for-byte — ruling out any client-side/deploy-tooling corruption. - Invoked the deployed agent. In every case:
- Application Insights received the trace correctly (confirmed via a Log Analytics query — full span tree,
gen_ai.*attributes present). - The third-party OTLP destination received nothing — no trace, no log, no metric — confirmed by querying that destination's own API directly for the exact trace ID.
- The container's own startup log line from
Azure.AI.AgentServer.Core.AgentHostBuilder—AgentServer connectivity: ProjectEndpoint=... OtlpEndpoint=(not set) AppInsightsConfigured=True— printedOtlpEndpoint=(not set)in every configuration tried, including the run whereOTEL_EXPORTER_OTLP_ENDPOINTalone was set and independently confirmed registered server-side.
- Application Insights received the trace correctly (confirmed via a Log Analytics query — full span tree,
- To rule out the destination itself, ran a minimal standalone .NET console app (
Microsoft.Extensions.Hosting+OpenTelemetry.Extensions.Hosting,AddOpenTelemetry().WithTracing(t => t.AddSource(...)).UseOtlpExporter()— the identical env-var-driven mechanism the doc describes), outside the hosted-agent runtime entirely, using the exact sameOTEL_EXPORTER_OTLP_PROTOCOL/OTEL_EXPORTER_OTLP_TRACES_ENDPOINT/OTEL_EXPORTER_OTLP_TRACES_HEADERSvalues already confirmed byte-correct in the deployed agent version. It delivered a real span to the third-party destination on the first try.
That isolation test rules out the destination, the API key, the header format, network reachability, and the .NET OpenTelemetry SDK's own OTLP exporter as causes. The defect is specific to the hosted-agent runtime's OpenTelemetry wiring: it never picks up any client-supplied third-party OTLP destination, while its separate Application Insights export path keeps working correctly throughout.
For context: an equivalent Python hosted agent, using the equivalent per-signal OTEL_EXPORTER_OTLP_* environment variables on the same underlying platform, does successfully deliver telemetry to the same third-party destination — so this appears to be specific to the .NET hosting stack, not a platform-wide limitation.
What I expected: Setting the documented OTEL_EXPORTER_OTLP_* environment variables on the agent version should activate OTLP export to that destination alongside Application Insights, with no code changes — as the platform doc describes.
Steps to reproduce:
- Create a .NET hosted agent using
AgentHost.CreateBuilder(args)+Microsoft.Agents.AI.Foundry.Hosting(see Code Sample — no manual OpenTelemetry instrumentation). - Deploy it to a Foundry project (
azure.yaml+azd deploy) with these env vars on the agent service:OTEL_EXPORTER_OTLP_ENDPOINT,OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, and/or the per-signalOTEL_EXPORTER_OTLP_{TRACES,METRICS,LOGS}_ENDPOINT/_HEADERSvariants, pointing at any OTLP-compliant third-party endpoint (reproduced against Datadog's agentless OTLP intake,https://otlp.datadoghq.com). - Invoke the deployed agent.
- Observe: Application Insights receives the trace; the third-party endpoint receives nothing; the container's startup log line reports
OtlpEndpoint=(not set)regardless of what was configured.
Code Sample
The entire agent — no manual telemetry code exists to inspect or misconfigure:
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Foundry.Hosting;
var builder = AgentHost.CreateBuilder(args);
AIAgent agent = /* built via AIProjectClient(...).AsAIAgent(...) */;
builder.Services.AddFoundryResponses(agent);
builder.RegisterProtocol("responses", endpoints => endpoints.MapFoundryResponses());
var app = builder.Build();
await app.RunAsync();
azure.yaml's relevant env block on the azure.ai.agent service (destination/keys genericized):
env:
OTEL_EXPORTER_OTLP_ENDPOINT: https://otlp.example-provider.com
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: https://otlp.example-provider.com/v1/traces
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT: https://otlp.example-provider.com/v1/metrics
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: https://otlp.example-provider.com/v1/logs
OTEL_EXPORTER_OTLP_HEADERS: api-key=<redacted>
The isolation repro that does work, run locally with the same env vars set in-process (not through the hosted-agent runtime):
using System.Diagnostics;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using OpenTelemetry;
using OpenTelemetry.Trace;
const string sourceName = "OtlpIsolationTest";
var builder = Host.CreateApplicationBuilder(args);
builder.Services.AddOpenTelemetry()
.WithTracing(tracing => tracing.AddSource(sourceName))
.UseOtlpExporter();
var host = builder.Build();
await host.StartAsync();
using var activitySource = new ActivitySource(sourceName);
using (activitySource.StartActivity("isolated-test-span")) { }
host.Services.GetRequiredService<TracerProvider>().ForceFlush(10000);
await host.StopAsync();
Error Messages / Stack Traces
info: Azure.AI.AgentServer.Core.AgentHostBuilder[0]
AgentServer connectivity: ProjectEndpoint=https://<account>.services.ai.azure.com OtlpEndpoint=(not set) AppInsightsConfigured=True
This line printed identically across every deployed agent version tested, regardless of which OTEL_EXPORTER_OTLP_* variables were set on that version (confirmed registered correctly server-side in every case via a GET on the agent version resource).
Package Versions
Microsoft.Agents.AI.Foundry.Hosting: 1.21.0-preview.260911.1 (pulls in Azure.AI.AgentServer.Core transitively)
.NET Version
.NET 10.0.11 (Linux container, dotnet_10 code-deploy runtime)
Additional Context
- Doc this behavior contradicts: Export hosted agent telemetry by using OpenTelemetry.
- This is not an Application Insights problem — that export path worked correctly in every single test in this investigation. It's specific to the third-party/OTLP path.
- This blocks any .NET hosted agent from sending telemetry to a non-Microsoft OpenTelemetry backend (Datadog, Honeycomb, a self-hosted Collector, etc.) via the documented, zero-code mechanism.
- Happy to share the full reproduction repo/branch privately if that's useful for triage.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at AgentHost.CreateBuilder and Azure.AI.AgentServer.Core.AgentHostBuilder, focusing on how the documented OTEL_EXPORTER_OTLP_* environment variables are read and how OtlpEndpoint is reported. Reproduce with Microsoft.Agents.AI.Foundry.Hosting 1.21.0-preview.260911.1 and the shown environment configuration, then verify that Application Insights remains functional while the third-party OTLP destination receives telemetry.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- backend, cloud, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100