Random failures regarding SegmentNotFoundException in multithreaded application
- Lingua principale
- Java
- Stelle
- 100
- Fork
- 100
- Metriche di merge delle PR
- Nessuna PR unita negli ultimi 30g
Descrizione
I am facing a problem in a multithreaded application where the dynamodb calls randomly fails to create a subsegment resulting in error message `"Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'AmazonDynamoDBv2': segment cannot be found."`
I am following the procedure in X-Ray documentation for passing segment context between threads as stated here - https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-multithreading.html and how its implemented in this example - https://docs.aws.amazon.com/xray/latest/devguide/scorekeep-workerthreads.html. In almost 70% cases the instrumentation is successful but the rest fails to find the segment when creating the subsegment.
In terms of implementation I am using custom fork join pool along with Java [parallel stream](https://docs.oracle.com/javase/8/docs/api/java/util/Collection.html#parallelStream--).
```
forkJoinPool
.submit(
() ->
filters.parallelStream()
.map(traceableFetcher)
.flatMap(Collection::stream)
.collect(Collectors.toList()))
.get();
```
`traceableFetcher` gets passed the segment using `awsXRayRecorder.getTraceEntity()` and the implementation of the traceable class is the following
```
public class TraceableFunction implements Function {
private final Function target;
private final Entity segment;
@Getter private final MdcSnapshot mdcSnapshot;
public TraceableFunction(Function target, final Entity segment) {
this.target = target;
this.segment = segment;
this.mdcSnapshot = new MdcSnapshot();
}
@Override
public R apply(T t) {
mdcSnapshot.populateMdc();
var resultReference = new AtomicReference();
segment.run(() -> resultReference.set(target.apply(t)));
mdcSnapshot.cleanMdc();
return resultReference.get();
}
}
```
It seems like in certain cases the [ThreadLocalStorage](https://github.com/aws/aws-xray-sdk-java/blob/d9d17ec980dce1c7e40a4b2e67cd5f76c5a36ea9/aws-xray-recorder-sdk-core/src/main/java/com/amazonaws/xray/ThreadLocalStorage.java#L27) doesn't have the segment or at least it has been cleared. Are you folks aware of any such issues or have any pointers? I am using `com.amazonaws:aws-xray-recorder-sdk-core:2.9.1`
Guida per i contributori
Apri la guida per i contributori
Direzione di ricerca
Inizia esaminando ThreadLocalStorage.java e l’implementazione di TraceableFunction, quindi riproduci il malfunzionamento con il pool fork-join personalizzato e l’esempio parallelStream usando aws-xray-recorder-sdk-core 2.9.1. Confronta le esecuzioni riuscite e quelle fallite per determinare quando viene perso il contesto del segmento; il lavoro è completato quando la SegmentNotFoundException intermittente è spiegata e l’instrumentazione multithread si comporta in modo coerente.
Scritto dal modello di indicizzazione a partire dal testo della issue.
Valutazione
- Stack tecnologico
- aws, java
- Ambito
- backend, distributed-systems, observability-sre
- Tipo di issue
- Bug
- Difficoltà
- 4/5
- Tempo stimato
- 3-5 giorni
- Stato di attività
- Ferma
- Chiarezza
- Abbastanza chiara
- Idoneità per principianti
- 32/100