elastic / elastic/apm-server

docs: Known issue on missing dependency in service map for Kong OpenTelemetry Plugin

Open
#16,446 0 comments 1 reaction 0 assignees View on GitHub
docs enhancement
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

Add documentation on the known issue about Kong OpenTelemetry Plugin leading to missing dependency link in APM Service Map.

# Description of problem

## To reproduce

I have a playground which is based on @1pkg 's playground: https://github.com/carsonip/kong-quickstart-otel/tree/carson

Tested with kong/kong-gateway:3.9.0.0

Setup: curl sends to service A behind Kong. Service A sends to B via Kong, and service B sends to C via Kong. See illustration

![Image](https://github.com/user-attachments/assets/49032ec1-ed2e-47a2-a0a5-00d2934a5ca9)

Result:
- resulting OTel docs from debugexporter: [collector.log](https://github.com/user-attachments/files/19487879/collector.log)
- docs indexed in ES: [docs.json](https://github.com/user-attachments/files/19487880/docs.json)

Highlight:

The exit span from Kong lacks required fields for APM to interpret it as a service dependency link
```
opentelemetry-collector-1 | Span #4
opentelemetry-collector-1 | Trace ID : bed5e28cfe42774e8ef1c9509685001d
opentelemetry-collector-1 | Parent ID : 052dc3538172140c
opentelemetry-collector-1 | ID : 3b8c984527a7f95e
opentelemetry-collector-1 | Name : kong.balancer
opentelemetry-collector-1 | Kind : Client
opentelemetry-collector-1 | Start time : 2025-03-27 10:03:09.028379392 +0000 UTC
opentelemetry-collector-1 | End time : 2025-03-27 10:03:09.030311168 +0000 UTC
opentelemetry-collector-1 | Status code : Unset
opentelemetry-collector-1 | Status message :
opentelemetry-collector-1 | Attributes:
opentelemetry-collector-1 | -> try_count: Double(1)
opentelemetry-collector-1 | -> net.peer.name: Str(172.17.0.1)
opentelemetry-collector-1 | -> net.peer.port: Double(10082)
opentelemetry-collector-1 | -> net.peer.ip: Str(172.17.0.1)
```

which gets translated to
```
{
"_index": ".ds-traces-apm-default-2025.03.27-000001",
"_id": "oscO15UBVFQppTdlRJwT",
"_score": 1,
"_source": {
"parent": {
"id": "052dc3538172140c"
},
"agent": {
"name": "otlp",
"version": "unknown"
},
"destination": {
"address": "172.17.0.1",
"ip": "172.17.0.1"
},
"processor": {
"event": "span"
},
"tags": [
"_geoip_database_unavailable_GeoLite2-City.mmdb"
],
"observer": {
"hostname": "e76267643b45",
"type": "apm-server",
"version": "8.17.4"
},
"trace": {
"id": "bed5e28cfe42774e8ef1c9509685001d"
},
"@timestamp": "2025-03-27T10:03:09.028Z",
"data_stream": {
"namespace": "default",
"type": "traces",
"dataset": "apm"
},
"numeric_labels": {
"net_peer_port": 10082,
"try_count": 1
},
"service": {
"node": {
"name": "f8636234-6a5d-4420-86df-4d89ec1c35a4"
},
"framework": {
"name": "kong-internal",
"version": "0.1.0"
},
"name": "kong-dev",
"language": {
"name": "unknown"
},
"version": "3.9.0.0"
},
"event": {
"ingested": "2025-03-27T10:03:11Z",
"success_count": 1,
"outcome": "success"
},
"span": {
"duration": {
"us": 1931
},
"representative_count": 1,
"name": "kong.balancer",
"id": "3b8c984527a7f95e",
"type": "unknown"
},
"timestamp": {
"us": 1743069789028379
}
}
}
```

Resulted in a service map that lacks every arrow from Kong to services:

![Image](https://github.com/user-attachments/assets/6ae2e3c1-cce3-4b6c-9b29-4754faf97792)

## Workaround

The goal of a workaround is to add the minimally required fields for APM to establish an arrow in service map.

Workaround is in commit https://github.com/carsonip/kong-quickstart-otel/tree/carson-workaround

The key is to add a transform processor to the trace pipeline of the otel collector that receives from kong. The processor will fill in `peer.service` from `net.peer.name` only for spans from kong balancer.

```
processors:
transform:
trace_statements:
- context: span
statements:
- set(attributes["peer.service"], attributes["net.peer.name"]) where attributes["net.peer.name"] != "" and name == "kong.balancer"
```

With that, the service map looks like this:

![Image](https://github.com/user-attachments/assets/d4fd5fc3-84ce-4cf4-a291-6822737fa3d3)

## Long term fix

Kong exit spans should follow OTel HTTP client span spec https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-client and Elastic APM should work out of the box. See https://github.com/Kong/kong/issues/14381

# Task

Update docs to mention the known issue briefly. Candidate location for docs: https://www.elastic.co/guide/en/observability/current/apm-open-telemetry-known-limitations.html

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.