aws / aws/aws-xray-sdk-java

Random failures regarding SegmentNotFoundException in multithreaded application

Aperta
#355 1 commento 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
100
Fork
100
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

I am facing a problem in a multithreaded application where the dynamodb calls randomly fails to create a subsegment resulting in error message `"Suppressing AWS X-Ray context missing exception (SegmentNotFoundException): Failed to begin subsegment named 'AmazonDynamoDBv2': segment cannot be found."`

I am following the procedure in X-Ray documentation for passing segment context between threads as stated here - https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-java-multithreading.html and how its implemented in this example - https://docs.aws.amazon.com/xray/latest/devguide/scorekeep-workerthreads.html. In almost 70% cases the instrumentation is successful but the rest fails to find the segment when creating the subsegment.

In terms of implementation I am using custom fork join pool along with Java [parallel stream](https://docs.oracle.com/javase/8/docs/api/java/util/Collection.html#parallelStream--).

```
forkJoinPool
.submit(
() ->
filters.parallelStream()
.map(traceableFetcher)
.flatMap(Collection::stream)
.collect(Collectors.toList()))
.get();
```

`traceableFetcher` gets passed the segment using `awsXRayRecorder.getTraceEntity()` and the implementation of the traceable class is the following

```
public class TraceableFunction implements Function {

private final Function target;

private final Entity segment;

@Getter private final MdcSnapshot mdcSnapshot;

public TraceableFunction(Function target, final Entity segment) {
this.target = target;
this.segment = segment;
this.mdcSnapshot = new MdcSnapshot();
}

@Override
public R apply(T t) {
mdcSnapshot.populateMdc();
var resultReference = new AtomicReference();

segment.run(() -> resultReference.set(target.apply(t)));

mdcSnapshot.cleanMdc();
return resultReference.get();
}
}
```

It seems like in certain cases the [ThreadLocalStorage](https://github.com/aws/aws-xray-sdk-java/blob/d9d17ec980dce1c7e40a4b2e67cd5f76c5a36ea9/aws-xray-recorder-sdk-core/src/main/java/com/amazonaws/xray/ThreadLocalStorage.java#L27) doesn't have the segment or at least it has been cleared. Are you folks aware of any such issues or have any pointers? I am using `com.amazonaws:aws-xray-recorder-sdk-core:2.9.1`

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Inizia esaminando ThreadLocalStorage.java e l’implementazione di TraceableFunction, quindi riproduci il malfunzionamento con il pool fork-join personalizzato e l’esempio parallelStream usando aws-xray-recorder-sdk-core 2.9.1. Confronta le esecuzioni riuscite e quelle fallite per determinare quando viene perso il contesto del segmento; il lavoro è completato quando la SegmentNotFoundException intermittente è spiegata e l’instrumentazione multithread si comporta in modo coerente.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, java
Ambito
backend, distributed-systems, observability-sre
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Abbastanza chiara
Idoneità per principianti
32/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.