Tracing a lambda subscribed to an SQS queue with propagated trace headers
- Dominant language
- Java
- Stars
- 100
- Forks
- 100
- PR merge metrics
- No merged PRs in 30d
Description
I've been banging my head against this for a while now and am not convinced it's even possible.
I have some X-ray traced code posting messages to an SQS queue that a Lambda is subscribed to, and I'm trying to link the Lambda subsegments to the trace coming through SQS. I'm following the guide [here](https://docs.aws.amazon.com/xray/latest/devguide/xray-services-sqs.html) but it doesn't seem to work at all within a Lambda.
If you do literally what the example code suggests there, you get a `NullPointerException` because there's no active segment when you call `AWSXRay.getCurrentSegment()` and the subsequent `setTraceId` and other methods on the return value fail.
I assumed the X-ray + Lambda machinery was going to create some sort of implicit subsegment for me that my API calls would be children of, and while that's true, it seems like it does it "on the fly" here:
https://github.com/aws/aws-xray-sdk-java/blob/c801c573054a7e20e1eb509937d5d880d329e181/aws-xray-recorder-sdk-core/src/main/java/com/amazonaws/xray/contexts/LambdaSegmentContext.java#L41-L47
by creating a `FacadeSegment` as the parent within each API call, so outside of my API call interceptor I don't have a segment or context to attach the SQS trace ID to.
So next I try to wrap my entire lambda logic in `AWSXray.createSegment` so I have a segment to attach the data to. But no, this code assumes that it's a `Subsegment` and gives me a `ClassCastException`:
https://github.com/aws/aws-xray-sdk-java/blob/c801c573054a7e20e1eb509937d5d880d329e181/aws-xray-recorder-sdk-core/src/main/java/com/amazonaws/xray/contexts/LambdaSegmentContext.java#L54
So then I try wrapping my entire lambda logic in `AWSXray.createSubsegment` to avoid the `ClassCastException`. That gets me a little farther, but now I need to do roughly this:
```java
segment.setTraceId(traceHeader.getRootTraceId());
segment.setParentId(traceHeader.getParentId());
segment.setSampled(traceHeader.getSampled().equals(TraceHeader.SampleDecision.SAMPLED));
```
And the first two calls work on my subsegment, but then the third `.setSampled` call fails first because I don't have a full `Segment`, and rather just a `Subsegment`, which doesn't support `setSampled`. If I reach into the parent and call `.getParent().setSampled`, I get hit by this awkward behavior in that `FacadeSegment`:
https://github.com/aws/aws-xray-sdk-java/blob/c183a3aad20a528a69cff26d77dfb691f503edb9/aws-xray-recorder-sdk-core/src/main/java/com/amazonaws/xray/entities/FacadeSegment.java#L102-L104
Indeed, it appears that the only way to set whether that `FacadeSegment` gets sampled is at construction time here:
https://github.com/aws/aws-xray-sdk-java/blob/c183a3aad20a528a69cff26d77dfb691f503edb9/aws-xray-recorder-sdk-core/src/main/java/com/amazonaws/xray/entities/FacadeSegment.java#L24-L29
But unfortunately I don't control its instantiation.
Even assuming the mechanics of all that wrangling work, which seems impossible given all the roadblocks I hit above, that would lose my Lambda-provided trace ID and replace it with the SQS trace ID, and I think I'd lose any metadata that the Lambda integration gives me. That seems inevitable unless I'm missing something.
Is there any sensible way to do this? It seems like both Lambda and SQS have first-class X-ray support but the combination of the two of them has been a huge exercise in frustration to me.
Contributor guide
Assessment
This issue has not been assessed yet.