Datadog tracer ignores Ingress/Egress — all spans get `span.kind:internal` (client/server metrics gone)
- Dominant language
- C++
- Stars
- 28.9k
- Forks
- 5.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 437
Description
*Title*: Datadog tracer ignores Ingress/Egress — all spans get `span.kind:internal` (client/server metrics gone)
*Description*:
After upgrading from Istio 1.17 (OpenTracing-based Datadog tracer) to Istio 1.30.2 (Envoy 1.38.x / `envoy.tracers.datadog` via dd-trace-cpp), Datadog APM metrics for Envoy proxy spans stopped splitting on `span.kind:client` and `span.kind:server`. All volume moved to `span.kind:internal`.
Observed (prod, kube_cluster_name=prod-use1-eks1):
| Window | client | server | internal |
|---|---|---|---|
| Pre-cutover (Istio 1.17, older Datadog tracer) | ~half | ~half | none |
| Post-cutover (Istio 1.30.2 / Envoy 1.38.3) | none | none | 100% |
Client/server series drop to zero at the cutover boundary; `internal` appears at the same time. Spans are still emitted correctly (`operation_name=envoy.proxy`, `component=proxy`, `language=cpp`), and direction is still visible via `upstream_cluster` (`inbound|…` vs `outbound|…`). Only the `span.kind` taxonomy is missing.
Expected: Envoy should set Datadog `span.kind` from traffic direction:
- `Tracing::OperationName::Ingress` → `span.kind=server`
- `Tracing::OperationName::Egress` → `span.kind=client`
Suspected root cause: in `source/extensions/tracers/datadog/tracer.cc` (`Tracer::startSpan`) and `span.cc` (`Span::spawnChild`), the driver explicitly ignores `Tracing::Config`:
// The OpenTracing implementation ignored the `Tracing::Config` argument,
// so we will as well.
`Tracing::Config::operationName()` is therefore never mapped onto the dd-trace-cpp span. With no `span.kind` tag, Datadog treats the span as `internal`.
Minimal fix — stop ignoring `config` and, after creating the span, set:
switch (config.operationName()) {
case Tracing::OperationName::Ingress:
span.set_tag("span.kind", "server");
break;
case Tracing::OperationName::Egress:
span.set_tag("span.kind", "client");
break;
}
(same in `spawnChild`), plus unit coverage that Ingress/Egress produce the corresponding `span.kind` meta.
This looks like a regression class from the OpenTracing → dd-trace-cpp rewrite (#26284), similar to the naming fallout fixed in #30235 / #31366. Happy to open a PR if maintainers agree on the approach. Please consider backport to the Envoy 1.38.x line (what Istio 1.30 ships).
cc @xlamorlette-datadog @zacharycmontoya @mattklein123
*Repro steps*:
1. Run Envoy 1.38.x (or an Istio 1.30.x sidecar) with the Datadog tracer enabled, e.g. via Istio Telemetry:
apiVersion: telemetry.istio.io/v1
kind: Telemetry
metadata:
name: mesh-logging
namespace: istio-system
spec:
tracing:
- providers:
- name: datadog
randomSamplingPercentage: 100
with `meshConfig.extensionProviders` pointing at a Datadog Agent on port 8126 (or equivalent bare-Envoy `envoy.tracers.datadog` config).
2. Generate both inbound (ingress) and outbound (egress) HTTP traffic.
3. In Datadog, graph: `sum:trace.envoy.proxy.hits{*} by {span.kind}.as_count()`
4. Observe only `span.kind:internal`. Inspect individual `operation_name:envoy.proxy` spans: no `span.kind` tag, while `upstream_cluster` still distinguishes `inbound|` vs `outbound|`. Contrast with Envoy ≤1.26, where the same metric split on `client`/`server`.
*Admin and Stats Output*:
Envoy version from live sidecars (Istio 1.30.2):
envoy_34ce4554c88740f88bfc4dc826ae33e1dd21d8fc/1.38.3-dev/clean/release/boringssl
Full `/stats` and `/clusters` omitted — not required to observe the tagging bug; can attach sanitized admin output if useful.
*Config*:
Istio mesh Datadog provider (sanitized):
meshConfig:
extensionProviders:
- name: datadog
datadog:
service: dd-agent.example.svc.cluster.local
port: 8126
Tracing driven by the Telemetry API (above). No `meshConfig.defaultConfig.tracing` set.
*Logs*:
N/A — spans export to the Datadog Agent successfully; the defect is missing `span.kind` on exported spans, not delivery failure.
*Call Stack*:
N/A (not a crash).
Contributor guide
Research direction
Start in source/extensions/tracers/datadog/tracer.cc at Tracer::startSpan and source/extensions/tracers/datadog/span.cc at Span::spawnChild, then inspect the existing Datadog tracer unit tests. Trace how Tracing::Config and its operationName are handled. Done means Ingress and Egress spans carry the expected server and client span.kind metadata, with unit coverage for both directions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 76/100