aws / aws/aws-xray-sdk-java

Random failures regarding SegmentNotFoundException in multithreaded application

Open
#355 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
100
Forks
100
PR merge metrics
No merged PRs in 30d

Description

I am facing a problem in a multithreaded application where the dynamodb calls randomly fails to create a subsegment resulting in error message `"Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'AmazonDynamoDBv2': segment cannot be found."`

I am following the procedure in X-Ray documentation for passing segment context between threads as stated here - https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-multithreading.html and how its implemented in this example - https://docs.aws.amazon.com/xray/latest/devguide/scorekeep-workerthreads.html. In almost 70% cases the instrumentation is successful but the rest fails to find the segment when creating the subsegment.

In terms of implementation I am using custom fork join pool along with Java [parallel stream](https://docs.oracle.com/javase/8/docs/api/java/util/Collection.html#parallelStream--).

```
forkJoinPool
.submit(
() ->
filters.parallelStream()
.map(traceableFetcher)
.flatMap(Collection::stream)
.collect(Collectors.toList()))
.get();
```

`traceableFetcher` gets passed the segment using `awsXRayRecorder.getTraceEntity()` and the implementation of the traceable class is the following

```
public class TraceableFunction implements Function {

private final Function target;

private final Entity segment;

@Getter private final MdcSnapshot mdcSnapshot;

public TraceableFunction(Function target, final Entity segment) {
this.target = target;
this.segment = segment;
this.mdcSnapshot = new MdcSnapshot();
}

@Override
public R apply(T t) {
mdcSnapshot.populateMdc();
var resultReference = new AtomicReference();

segment.run(() -> resultReference.set(target.apply(t)));

mdcSnapshot.cleanMdc();
return resultReference.get();
}
}
```

It seems like in certain cases the [ThreadLocalStorage](https://github.com/aws/aws-xray-sdk-java/blob/d9d17ec980dce1c7e40a4b2e67cd5f76c5a36ea9/aws-xray-recorder-sdk-core/src/main/java/com/amazonaws/xray/ThreadLocalStorage.java#L27) doesn't have the segment or at least it has been cleared. Are you folks aware of any such issues or have any pointers? I am using `com.amazonaws:aws-xray-recorder-sdk-core:2.9.1`

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.