aws / aws/aws-xray-sdk-node

Segment from Lambda triggered by SQS is never sampled

Open
#570 3 comments 1 reaction 0 assignees View on GitHub
stale
Dominant language
JavaScript
Stars
280
Forks
157
PR merge metrics
No merged PRs in 30d

Description

Hey folks, I've been scratching my head against this one for a while, I created this issue hoping that someone can help figure out what's going on.

I have a Lambda function with an alias, and the alias can be triggered by SQS. Events are coming in from EventBridge and routed to SQS. The function has Active tracing enabled.

When the function gets triggered by SQS, I see the following in my logs (values from one specific run are shown below, but this is consistent behaviour on every attempt):
- The SQS message itself contains the following in its attributes:
```
"AWSTraceHeader": "Root=1-63e54f60-54c0dec03aa822cf0eb80f6b;Parent=7026c82d12761ec6;Sampled=0"
```
- The X-Ray SDK ends up apparently ignoring that though, and uses this facade segment:
```
Lambda trace data found: Root=1-63e54f76-613d90840ef5611617d19f78;Parent=5b3f323a471e441f;Sampled=0
Segment started:
{
"root": "1-63e54f76-613d90840ef5611617d19f78",
"parent": "5b3f323a471e441f",
"sampled": "0",
"data": {}
}
```
- At the end of the function, the SDK ends up deciding not to flush any subsegments, because (as I showed), the segment is not sampled:
```
Ignoring flush on subsegment 3b13dabf96025041. Associated segment is marked as not sampled.
```

Note how both of those segments are not sampled (the one from SQS and the one Lambda creates). I've tried recreating the whole infrastructure (SQS queues, EventBridge triggers, Lambda function, IAM roles) from scratch, and I tried disabling Active tracing and re-enabling it, but this behaviour keeps happening. The sample policy I'm using is the default one, and the Lambda was invoked only to test this, which means it was definitely within the policy of 1 sample per second.

However, if I manually invoke the Lambda function (with the same message it gets from SQS, no changes at all, including the unsampled `AWSTraceHeader`), then it creates a sampled segment (as expected!). Note that this is a manual invocation of the Lambda, it isn't being triggered by SQS this time:
```
Lambda trace data found: Root=1-63e5ac97-07c68b9b2d943eb56a158f57;Parent=e42fbe5a5a09d152;Sampled=1
Segment started:
{
"root": "1-63e5ac97-07c68b9b2d943eb56a158f57",
"parent": "e42fbe5a5a09d152",
"sampled": "1",
"data": {}
}
```

And it properly flushes the subsegments, and I can see them in the X-Ray console:
```
Subsegment sent: {"trace_id:"1-63e5ac97-07c68b9b2d943eb56a158f57","id":"b2f07bfe66ccf783"}
UDP message sent:
{
"id": "b2f07bfe66ccf783",
"name": "",
"start_time": 1675996312.821,
"namespace": "remote",
"http": {

},
"end_time": 1675996313.035,
"type": "subsegment",
"parent_id": "e42fbe5a5a09d152",
"trace_id": "1-63e5ac97-07c68b9b2d943eb56a158f57"
}
```

Does anyone know why apparently the Lambda function is never sampled when invoked through the SQS trigger? I noticed you folks have recently worked on an SQS->Lambda and SNS->SQS->Lambda trace continuation features (congrats btw! I'm definitely going to make use of them), so I'm wondering if something along the way changed and broke this scenario I'm describing?

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the SQS-triggered Lambda path and compare its trace continuation with the manual invocation described in the issue. Inspect the recent SQS→Lambda and SNS→SQS→Lambda trace-continuation changes, then add or run a regression test showing that a qualifying invocation produces a sampled segment and flushes its subsegments.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
cloud, distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.