aws / aws/aws-xray-sdk-python

Trace Size Exceeded

Aperta
#309 4 commenti 0 reazioni 0 assegnatari Vedi su GitHub
question stale
Lingua principale
Python
Stelle
338
Fork
147
Metriche di merge delle PR
Nessuna PR unita negli ultimi 30g

Descrizione

Hi,

I am starting to see this error a lot in most of our django applications.

```
[Error] Unprocessed segment: {
ErrorCode: "InvalidSegment",
Id: "xxxxxxxxxxxxxxxxxx",
Message: "Trace Size Exceeded. Reduce the number of segments per trace and the size of your segment documents."
}
```

Also, whilst looking at the logs:
```
2021-07-21T11:37:15.068+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.016 seconds)
2021-07-21T11:37:15.071+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.087+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.102+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.117+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.135+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.007 seconds)
2021-07-21T11:37:15.148+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.163+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.178+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.194+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.214+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.225+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.239+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.256+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.288+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.017 seconds)
2021-07-21T11:37:15.289+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.311+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.326+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.336+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.350+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.365+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.379+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.395+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.415+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.424+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.439+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.458+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.009 seconds)
2021-07-21T11:37:15.467+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.483+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.497+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.513+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.534+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.554+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.011 seconds)
2021-07-21T11:37:15.563+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.577+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.593+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.610+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.007 seconds)
2021-07-21T11:37:15.624+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.641+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.006 seconds)
```
It looks like a lot of segments are being sent while nothing is really happening on the django application itself.

We are running the daemon on the alongside the django application, in the same docker container. We have the recorder setup as instructed [here](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-python-configuration.html#xray-sdk-python-middleware-configuration-django):
```
XRAY_RECORDER = {
"AUTO_INSTRUMENT": True,
"AWS_XRAY_CONTEXT_MISSING": "LOG_ERROR",
"AWS_XRAY_TRACING_NAME": f"{env('PARTNER', default='embed')}-{env('ENVIRONMENT_TYPE')}",
"PLUGINS": ("EC2Plugin", "ECSPlugin"),
"AWS_XRAY_SDK_ENABLED": env("AWS_XRAY_SDK_ENABLED", default=False),
}
```
I am also seeing this when the application starts:
```
[INFO] No effective centralized sampling rule match. Fallback to local rules.
```
Our sampling rules are not local tho, but defined in the xray we console.

Guida per i contributori

Apri la guida per i contributori

Direzione di ricerca

Iniziare dalla configurazione di Django recorder e middleware mostrata nell’issue e dalla documentazione sulla configurazione di AWS X-Ray Python a cui rimanda. Esaminare le voci di log ripetute da 50 segmenti del daemon insieme al messaggio di fallback del campionamento centralizzato; il lavoro è completato quando viene identificata la causa dell’errore relativo alle dimensioni della trace e viene documentata una risoluzione attuabile.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
aws, django, docker, python
Ambito
observability-sre
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Ferma
Chiarezza
Da chiarire
Idoneità per principianti
28/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.