elastic / elastic/apm-server

APM service map assumes service.environment is null for some services, possibly causing missing links in service map.

Open
#8,254 7 comments 1 reaction 2 assignees Claimed by @kruskall View on GitHub
apm:service-maps bug OpenTelemetry
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

Cross-posting from this [discussion board thread](https://discuss.elastic.co/t/troubleshooting-missing-links-for-some-opentelemetry-services-in-the-apm-service-map/302451) which I now believe could be a bug.

**Elastic version:** 8.1.2

**Elastic environment:** Elastic Cloud on GCP us-east1

**APM instrumentation**: OpenTelemetry

**Client browser:** Chrome

**Client OS:** MacOS 12.1

**Describe the bug:** The APM Service Map incorrectly sets `service.environment` to `null` for two of my services, which could might be the reason why they appear [orphaned in the service map](https://discuss.elastic.co/t/troubleshooting-missing-links-for-some-opentelemetry-services-in-the-apm-service-map/302451). I've verified that the spans for those services do have `service.environment` set to `development` in every span that references those services. This behavior is happening consistently for only those two services, even after starting with a fresh dataset and cluster. Each Python service is instrumented using the same code and the same environment variables (`OTEL_RESOURCE_ATTRIBUTES` set to `deployment.environment=development`), and therefore they all should behave very similarly for tracing.

**Steps to reproduce:** This is an instance of [microbs](https://github.com/microbs-io/microbs) ecommerce application. It might be easier to troubleshoot if I provided direct access to the Elastic Cloud deployment where the APM data resides, because the deployment does not have sensitive data.

**Response from `GET https://ELASTICSEARCH_ENDPOINT/.ds-traces-apm*/_search?q=(service.name:payment+OR+service.name:product)+AND+NOT+service.environment:development`** - Observe that there are no spans for the payment or product service in which `service.environment` is not `development`.

```
{
"took" : 85,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
```

**Response from `GET https://KIBANA_ENDPOINT/internal/apm/service-map`** - Observe that `service.environment` is `null` for the payment and product services, which is inconsistent with the results of the prior query.

```
{
"elements": [{
"data": {
"id": "web-gateway",
"service.environment": "development",
"service.name": "web-gateway",
"agent.name": "opentelemetry/python"
}
}, {
"data": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
}
}, {
"data": {
"id": "content",
"service.environment": "development",
"service.name": "content",
"agent.name": "opentelemetry/python"
}
}, {
"data": {
"span.subtype": "http",
"span.destination.service.resource": "storage.googleapis.com:443",
"span.type": "external",
"id": ">storage.googleapis.com:443",
"label": "storage.googleapis.com:443"
}
}, {
"data": {
"id": "checkout",
"service.environment": "development",
"service.name": "checkout",
"agent.name": "opentelemetry/python"
}
}, {
"data": {
"id": "cart",
"service.environment": "development",
"service.name": "cart",
"agent.name": "opentelemetry/python"
}
}, {
"data": {
"span.subtype": "redis",
"span.destination.service.resource": "redis",
"span.type": "db",
"id": ">redis",
"label": "redis"
}
}, {
"data": {
"service.name": "product",
"agent.name": "opentelemetry/python",
"service.environment": null,
"id": "product"
}
}, {
"data": {
"service.name": "payment",
"agent.name": "opentelemetry/python",
"service.environment": null,
"id": "payment"
}
}, {
"data": {
"source": "api-gateway",
"target": "cart",
"id": "api-gateway~cart",
"sourceData": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
},
"targetData": {
"id": "cart",
"service.environment": "development",
"service.name": "cart",
"agent.name": "opentelemetry/python"
}
}
}, {
"data": {
"source": "api-gateway",
"target": "checkout",
"id": "api-gateway~checkout",
"sourceData": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
},
"targetData": {
"id": "checkout",
"service.environment": "development",
"service.name": "checkout",
"agent.name": "opentelemetry/python"
},
"bidirectional": true
}
}, {
"data": {
"source": "api-gateway",
"target": "content",
"id": "api-gateway~content",
"sourceData": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
},
"targetData": {
"id": "content",
"service.environment": "development",
"service.name": "content",
"agent.name": "opentelemetry/python"
}
}
}, {
"data": {
"source": "cart",
"target": ">redis",
"id": "cart~>redis",
"sourceData": {
"id": "cart",
"service.environment": "development",
"service.name": "cart",
"agent.name": "opentelemetry/python"
},
"targetData": {
"span.subtype": "redis",
"span.destination.service.resource": "redis",
"span.type": "db",
"id": ">redis",
"label": "redis"
}
}
}, {
"data": {
"source": "checkout",
"target": "api-gateway",
"id": "checkout~api-gateway",
"sourceData": {
"id": "checkout",
"service.environment": "development",
"service.name": "checkout",
"agent.name": "opentelemetry/python"
},
"targetData": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
},
"isInverseEdge": true
}
}, {
"data": {
"source": "content",
"target": ">storage.googleapis.com:443",
"id": "content~>storage.googleapis.com:443",
"sourceData": {
"id": "content",
"service.environment": "development",
"service.name": "content",
"agent.name": "opentelemetry/python"
},
"targetData": {
"span.subtype": "http",
"span.destination.service.resource": "storage.googleapis.com:443",
"span.type": "external",
"id": ">storage.googleapis.com:443",
"label": "storage.googleapis.com:443"
}
}
}, {
"data": {
"source": "web-gateway",
"target": "api-gateway",
"id": "web-gateway~api-gateway",
"sourceData": {
"id": "web-gateway",
"service.environment": "development",
"service.name": "web-gateway",
"agent.name": "opentelemetry/python"
},
"targetData": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
}
}
}]
}
```

**Screenshot 1 of 2** - The service map is missing links among the product and payment services, whose `service.environment` is set to `null` in the XHR response with the service map data. Note that `service.environment` is actually set to `development` in all of the spans for those two services, which I confirmed by searching in Discover.

service-map

**Screenshot 2 of 2** - This trace sample does display links between the services that were unlinked in the service map. This screenshot shows that the payment service is linked to the api-gateway service, but that link doesn't appear in the service map.

trace

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.