APM service map assumes service.environment is null for some services, possibly causing missing links in service map.
- Dominant language
- Go
- Stars
- 1.3k
- Forks
- 543
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 109
Description
Cross-posting from this [discussion board thread](https://discuss.elastic.co/t/troubleshooting-missing-links-for-some-opentelemetry-services-in-the-apm-service-map/302451) which I now believe could be a bug.
**Elastic version:** 8.1.2
**Elastic environment:** Elastic Cloud on GCP us-east1
**APM instrumentation**: OpenTelemetry
**Client browser:** Chrome
**Client OS:** MacOS 12.1
**Describe the bug:** The APM Service Map incorrectly sets `service.environment` to `null` for two of my services, which could might be the reason why they appear [orphaned in the service map](https://discuss.elastic.co/t/troubleshooting-missing-links-for-some-opentelemetry-services-in-the-apm-service-map/302451). I've verified that the spans for those services do have `service.environment` set to `development` in every span that references those services. This behavior is happening consistently for only those two services, even after starting with a fresh dataset and cluster. Each Python service is instrumented using the same code and the same environment variables (`OTEL_RESOURCE_ATTRIBUTES` set to `deployment.environment=development`), and therefore they all should behave very similarly for tracing.
**Steps to reproduce:** This is an instance of [microbs](https://github.com/microbs-io/microbs) ecommerce application. It might be easier to troubleshoot if I provided direct access to the Elastic Cloud deployment where the APM data resides, because the deployment does not have sensitive data.
**Response from `GET https://ELASTICSEARCH_ENDPOINT/.ds-traces-apm*/_search?q=(service.name:payment+OR+service.name:product)+AND+NOT+service.environment:development`** - Observe that there are no spans for the payment or product service in which `service.environment` is not `development`.
```
{
"took" : 85,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
```
**Response from `GET https://KIBANA_ENDPOINT/internal/apm/service-map`** - Observe that `service.environment` is `null` for the payment and product services, which is inconsistent with the results of the prior query.
```
{
"elements": [{
"data": {
"id": "web-gateway",
"service.environment": "development",
"service.name": "web-gateway",
"agent.name": "opentelemetry/python"
}
}, {
"data": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
}
}, {
"data": {
"id": "content",
"service.environment": "development",
"service.name": "content",
"agent.name": "opentelemetry/python"
}
}, {
"data": {
"span.subtype": "http",
"span.destination.service.resource": "storage.googleapis.com:443",
"span.type": "external",
"id": ">storage.googleapis.com:443",
"label": "storage.googleapis.com:443"
}
}, {
"data": {
"id": "checkout",
"service.environment": "development",
"service.name": "checkout",
"agent.name": "opentelemetry/python"
}
}, {
"data": {
"id": "cart",
"service.environment": "development",
"service.name": "cart",
"agent.name": "opentelemetry/python"
}
}, {
"data": {
"span.subtype": "redis",
"span.destination.service.resource": "redis",
"span.type": "db",
"id": ">redis",
"label": "redis"
}
}, {
"data": {
"service.name": "product",
"agent.name": "opentelemetry/python",
"service.environment": null,
"id": "product"
}
}, {
"data": {
"service.name": "payment",
"agent.name": "opentelemetry/python",
"service.environment": null,
"id": "payment"
}
}, {
"data": {
"source": "api-gateway",
"target": "cart",
"id": "api-gateway~cart",
"sourceData": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
},
"targetData": {
"id": "cart",
"service.environment": "development",
"service.name": "cart",
"agent.name": "opentelemetry/python"
}
}
}, {
"data": {
"source": "api-gateway",
"target": "checkout",
"id": "api-gateway~checkout",
"sourceData": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
},
"targetData": {
"id": "checkout",
"service.environment": "development",
"service.name": "checkout",
"agent.name": "opentelemetry/python"
},
"bidirectional": true
}
}, {
"data": {
"source": "api-gateway",
"target": "content",
"id": "api-gateway~content",
"sourceData": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
},
"targetData": {
"id": "content",
"service.environment": "development",
"service.name": "content",
"agent.name": "opentelemetry/python"
}
}
}, {
"data": {
"source": "cart",
"target": ">redis",
"id": "cart~>redis",
"sourceData": {
"id": "cart",
"service.environment": "development",
"service.name": "cart",
"agent.name": "opentelemetry/python"
},
"targetData": {
"span.subtype": "redis",
"span.destination.service.resource": "redis",
"span.type": "db",
"id": ">redis",
"label": "redis"
}
}
}, {
"data": {
"source": "checkout",
"target": "api-gateway",
"id": "checkout~api-gateway",
"sourceData": {
"id": "checkout",
"service.environment": "development",
"service.name": "checkout",
"agent.name": "opentelemetry/python"
},
"targetData": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
},
"isInverseEdge": true
}
}, {
"data": {
"source": "content",
"target": ">storage.googleapis.com:443",
"id": "content~>storage.googleapis.com:443",
"sourceData": {
"id": "content",
"service.environment": "development",
"service.name": "content",
"agent.name": "opentelemetry/python"
},
"targetData": {
"span.subtype": "http",
"span.destination.service.resource": "storage.googleapis.com:443",
"span.type": "external",
"id": ">storage.googleapis.com:443",
"label": "storage.googleapis.com:443"
}
}
}, {
"data": {
"source": "web-gateway",
"target": "api-gateway",
"id": "web-gateway~api-gateway",
"sourceData": {
"id": "web-gateway",
"service.environment": "development",
"service.name": "web-gateway",
"agent.name": "opentelemetry/python"
},
"targetData": {
"id": "api-gateway",
"service.name": "api-gateway",
"agent.name": "opentelemetry/cpp"
}
}
}]
}
```
**Screenshot 1 of 2** - The service map is missing links among the product and payment services, whose `service.environment` is set to `null` in the XHR response with the service map data. Note that `service.environment` is actually set to `development` in all of the spans for those two services, which I confirmed by searching in Discover.

**Screenshot 2 of 2** - This trace sample does display links between the services that were unlinked in the service map. This screenshot shows that the payment service is linked to the api-gateway service, but that link doesn't appear in the service map.

Contributor guide
Assessment
This issue has not been assessed yet.