elastic / elastic/apm

APM UI doesn't show connected traces sent from AWS Distro for OTel (Lambda)

Open
#458 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
Gherkin
Stars
427
Forks
125
PR merge metrics
No merged PRs in 30d

Description

**APM Server version** (`apm-server version`): 7.13.1

**Description of the problem including expected versus actual behavior**:
I managed to get the distributed tracing going with AWS X-Ray and the OTel collector to pick it up. The flow is `curl => API-GW => producer-lambda => API-GW => consumer-lambda`. The code (serverless): https://github.com/michaelhyatt/serverless-aws-python-api-worker-otel

The interesting thing is that we are recording the 2 transactions with 1 span correctly as having the same `trace.id`, however, the APM UI doesn’t really show a proper trace.

AWS X-Ray showing the trace with 2 components:
![image](https://user-images.githubusercontent.com/15670925/123197589-0de02e00-d4ef-11eb-91b1-017e5b925b59.png)

Discover shows 3 records, 2 transactions and 1 span with the same `trace.id`:
![image](https://user-images.githubusercontent.com/15670925/123197659-30724700-d4ef-11eb-9c25-93e71632bfb2.png)

The APM UI is showing 2 services, but no traces:
![image](https://user-images.githubusercontent.com/15670925/123197775-6283a900-d4ef-11eb-96b7-380556db0e95.png)

**Steps to reproduce**:
As per README, requires the serverless framework.

JSON documents

### Consumer transaction
```json
{
"_index": "apm-7.13.1-transaction-000001",
"_type": "_doc",
"_id": "TdX7OHoBmWidblu2OYlN",
"_version": 1,
"_score": null,
"fields": {
"transaction.name.text": ["handler.consumer"],
"service.framework.version": ["0.16.dev0"],
"observer.name": ["instance-0000000008"],
"labels.faas_name": ["aws-python-api-worker-project-dev-consumer"],
"service.language.name": ["python"],
"labels.faas_id": ["arn:aws:lambda:ap-southeast-2:401722391821:function:aws-python-api-worker-project-dev-consumer"],
"transaction.sampled": [true],
"transaction.id": ["1b5693f6e07de9c7"],
"trace.id": ["60d331f41b8d383766d158e55359aaca"],
"processor.event": ["transaction"],
"agent.name": ["opentelemetry/python"],
"labels.faas_execution": ["c7ab9723-9e43-4488-89d0-e1759190b984"],
"event.outcome": ["unknown"],
"cloud.region": ["ap-southeast-2"],
"service.name": ["aws-python-api-worker-project-dev-consumer"],
"service.framework.name": ["opentelemetry.instrumentation.aws_lambda"],
"processor.name": ["transaction"],
"transaction.duration.us": [698],
"labels.faas_version": ["$LATEST"],
"observer.version_major": [7],
"observer.hostname": ["60e5fd4e9651"],
"transaction.type": ["custom"],
"cloud.provider": ["aws"],
"event.ingested": ["2021-06-23T13:07:06.188Z"],
"timestamp.us": [1624453624947661],
"@timestamp": ["2021-06-23T13:07:04.947Z"],
"ecs.version": ["1.8.0"],
"observer.type": ["apm-server"],
"observer.version": ["7.13.1"],
"agent.version": ["1.2.0"],
"parent.id": ["8ae9f83b97976607"],
"transaction.name": ["handler.consumer"]
},
"highlight": {
"trace.id": ["@kibana-highlighted-field@60d331f41b8d383766d158e55359aaca@/kibana-highlighted-field@"]
},
"sort": [1624453624947]
}
```
### Producer HTTP span
```json
{
"_index": "apm-7.13.1-span-000001",
"_type": "_doc",
"_id": "T9X7OHoBmWidblu2OYlN",
"_version": 1,
"_score": null,
"fields": {
"span.destination.service.type": ["external"],
"span.name": ["HTTP GET"],
"destination.port": [443],
"observer.name": ["instance-0000000008"],
"labels.faas_name": ["aws-python-api-worker-project-dev-producer"],
"trace.id": ["60d331f41b8d383766d158e55359aaca"],
"span.duration.us": [2256175],
"processor.event": ["span"],
"agent.name": ["opentelemetry/python"],
"destination.address": ["3y00jobl0c.execute-api.ap-southeast-2.amazonaws.com"],
"event.outcome": ["success"],
"cloud.region": ["ap-southeast-2"],
"service.name": ["aws-python-api-worker-project-dev-producer"],
"processor.name": ["transaction"],
"labels.faas_version": ["$LATEST"],
"span.id": ["898690921573f428"],
"span.subtype": ["http"],
"observer.version_major": [7],
"observer.hostname": ["60e5fd4e9651"],
"span.type": ["external"],
"cloud.provider": ["aws"],
"timestamp.us": [1624453622779541],
"@timestamp": ["2021-06-23T13:07:02.779Z"],
"ecs.version": ["1.8.0"],
"observer.type": ["apm-server"],
"observer.version": ["7.13.1"],
"agent.version": ["1.2.0"],
"parent.id": ["8b8e0c642d96e5f2"],
"span.destination.service.name": ["https://3y00jobl0c.execute-api.ap-southeast-2.amazonaws.com"],
"span.destination.service.resource": ["3y00jobl0c.execute-api.ap-southeast-2.amazonaws.com:443"]
},
"highlight": {
"trace.id": ["@kibana-highlighted-field@60d331f41b8d383766d158e55359aaca@/kibana-highlighted-field@"]
},
"sort": [1624453622779]
}
```
### Producer transaction
```json
{
"_index": "apm-7.13.1-transaction-000001",
"_type": "_doc",
"_id": "TtX7OHoBmWidblu2OYlN",
"_version": 1,
"_score": null,
"fields": {
"transaction.name.text": ["handler.producer"],
"service.framework.version": ["0.21b0"],
"observer.name": ["instance-0000000008"],
"labels.faas_name": ["aws-python-api-worker-project-dev-producer"],
"service.language.name": ["python"],
"labels.faas_id": ["arn:aws:lambda:ap-southeast-2:401722391821:function:aws-python-api-worker-project-dev-producer"],
"transaction.sampled": [true],
"transaction.id": ["8b8e0c642d96e5f2"],
"trace.id": ["60d331f41b8d383766d158e55359aaca"],
"processor.event": ["transaction"],
"agent.name": ["opentelemetry/python"],
"labels.faas_execution": ["fdcc0a5f-ce57-4201-a631-6097957f4a78"],
"event.outcome": ["unknown"],
"cloud.region": ["ap-southeast-2"],
"service.name": ["aws-python-api-worker-project-dev-producer"],
"service.framework.name": ["opentelemetry.instrumentation.requests"],
"processor.name": ["transaction"],
"transaction.duration.us": [2259030],
"labels.faas_version": ["$LATEST"],
"observer.version_major": [7],
"observer.hostname": ["60e5fd4e9651"],
"transaction.type": ["custom"],
"cloud.provider": ["aws"],
"event.ingested": ["2021-06-23T13:07:06.189Z"],
"timestamp.us": [1624453622778742],
"@timestamp": ["2021-06-23T13:07:02.778Z"],
"ecs.version": ["1.8.0"],
"observer.type": ["apm-server"],
"observer.version": ["7.13.1"],
"agent.version": ["1.2.0"],
"parent.id": ["918243f3ba6e3f71"],
"transaction.name": ["handler.producer"]
},
"highlight": {
"trace.id": ["@kibana-highlighted-field@60d331f41b8d383766d158e55359aaca@/kibana-highlighted-field@"]
},
"sort": [1624453622778]
}
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.