Segment from Lambda triggered by SQS is never sampled
- Dominant language
- JavaScript
- Stars
- 280
- Forks
- 157
- PR merge metrics
- No merged PRs in 30d
Description
Hey folks, I've been scratching my head against this one for a while, I created this issue hoping that someone can help figure out what's going on.
I have a Lambda function with an alias, and the alias can be triggered by SQS. Events are coming in from EventBridge and routed to SQS. The function has Active tracing enabled.
When the function gets triggered by SQS, I see the following in my logs (values from one specific run are shown below, but this is consistent behaviour on every attempt):
- The SQS message itself contains the following in its attributes:
```
"AWSTraceHeader": "Root=1-63e54f60-54c0dec03aa822cf0eb80f6b;Parent=7026c82d12761ec6;Sampled=0"
```
- The X-Ray SDK ends up apparently ignoring that though, and uses this facade segment:
```
Lambda trace data found: Root=1-63e54f76-613d90840ef5611617d19f78;Parent=5b3f323a471e441f;Sampled=0
Segment started:
{
"root": "1-63e54f76-613d90840ef5611617d19f78",
"parent": "5b3f323a471e441f",
"sampled": "0",
"data": {}
}
```
- At the end of the function, the SDK ends up deciding not to flush any subsegments, because (as I showed), the segment is not sampled:
```
Ignoring flush on subsegment 3b13dabf96025041. Associated segment is marked as not sampled.
```
Note how both of those segments are not sampled (the one from SQS and the one Lambda creates). I've tried recreating the whole infrastructure (SQS queues, EventBridge triggers, Lambda function, IAM roles) from scratch, and I tried disabling Active tracing and re-enabling it, but this behaviour keeps happening. The sample policy I'm using is the default one, and the Lambda was invoked only to test this, which means it was definitely within the policy of 1 sample per second.
However, if I manually invoke the Lambda function (with the same message it gets from SQS, no changes at all, including the unsampled `AWSTraceHeader`), then it creates a sampled segment (as expected!). Note that this is a manual invocation of the Lambda, it isn't being triggered by SQS this time:
```
Lambda trace data found: Root=1-63e5ac97-07c68b9b2d943eb56a158f57;Parent=e42fbe5a5a09d152;Sampled=1
Segment started:
{
"root": "1-63e5ac97-07c68b9b2d943eb56a158f57",
"parent": "e42fbe5a5a09d152",
"sampled": "1",
"data": {}
}
```
And it properly flushes the subsegments, and I can see them in the X-Ray console:
```
Subsegment sent: {"trace_id:"1-63e5ac97-07c68b9b2d943eb56a158f57","id":"b2f07bfe66ccf783"}
UDP message sent:
{
"id": "b2f07bfe66ccf783",
"name": "",
"start_time": 1675996312.821,
"namespace": "remote",
"http": {
},
"end_time": 1675996313.035,
"type": "subsegment",
"parent_id": "e42fbe5a5a09d152",
"trace_id": "1-63e5ac97-07c68b9b2d943eb56a158f57"
}
```
Does anyone know why apparently the Lambda function is never sampled when invoked through the SQS trigger? I noticed you folks have recently worked on an SQS->Lambda and SNS->SQS->Lambda trace continuation features (congrats btw! I'm definitely going to make use of them), so I'm wondering if something along the way changed and broke this scenario I'm describing?
Contributor guide
Research direction
Start by reproducing the SQS-triggered Lambda path and compare its trace continuation with the manual invocation described in the issue. Inspect the recent SQS→Lambda and SNS→SQS→Lambda trace-continuation changes, then add or run a regression test showing that a qualifying invocation produces a sampled segment and flushes its subsegments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- cloud, distributed-systems, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100