Trace Size Exceeded
- Langage dominant
- Python
- Étoiles
- 339
- Forks
- 147
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
Hi,
I am starting to see this error a lot in most of our django applications.
```
[Error] Unprocessed segment: {
ErrorCode: "InvalidSegment",
Id: "xxxxxxxxxxxxxxxxxx",
Message: "Trace Size Exceeded. Reduce the number of segments per trace and the size of your segment documents."
}
```
Also, whilst looking at the logs:
```
2021-07-21T11:37:15.068+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.016 seconds)
2021-07-21T11:37:15.071+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.087+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.102+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.117+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.135+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.007 seconds)
2021-07-21T11:37:15.148+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.163+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.178+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.194+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.214+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.225+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.239+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.256+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.288+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.017 seconds)
2021-07-21T11:37:15.289+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.311+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.326+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.336+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.350+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.365+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.379+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.395+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.415+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.424+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.439+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.458+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.009 seconds)
2021-07-21T11:37:15.467+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.483+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.497+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.513+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.534+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.554+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.011 seconds)
2021-07-21T11:37:15.563+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.577+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.593+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.610+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.007 seconds)
2021-07-21T11:37:15.624+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.641+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.006 seconds)
```
It looks like a lot of segments are being sent while nothing is really happening on the django application itself.
We are running the daemon on the alongside the django application, in the same docker container. We have the recorder setup as instructed [here](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-python-configuration.html#xray-sdk-python-middleware-configuration-django):
```
XRAY_RECORDER = {
"AUTO_INSTRUMENT": True,
"AWS_XRAY_CONTEXT_MISSING": "LOG_ERROR",
"AWS_XRAY_TRACING_NAME": f"{env('PARTNER', default='embed')}-{env('ENVIRONMENT_TYPE')}",
"PLUGINS": ("EC2Plugin", "ECSPlugin"),
"AWS_XRAY_SDK_ENABLED": env("AWS_XRAY_SDK_ENABLED", default=False),
}
```
I am also seeing this when the application starts:
```
[INFO] No effective centralized sampling rule match. Fallback to local rules.
```
Our sampling rules are not local tho, but defined in the xray we console.
Guide de contribution
Ouvrir le guide de contribution
Piste de recherche
Commencez par la configuration de Django recorder et middleware présentée dans l’issue, ainsi que par la documentation de configuration AWS X-Ray Python vers laquelle elle renvoie. Examinez les entrées de journal répétées de 50 segments du daemon avec le message de fallback de l’échantillonnage centralisé ; le travail est terminé lorsque la cause de l’erreur de taille de trace est identifiée et qu’une résolution exploitable est documentée.
Rédigé par le modèle d'indexation à partir du texte de l'issue.
Évaluation
- Stack technique
- aws, django, docker, python
- Domaine
- observability-sre
- Type d'issue
- Bug
- Difficulté
- 4/5
- Temps estimé
- 3-5 jours
- Activité
- À l'abandon
- Clarté
- À clarifier
- Accessibilité débutants
- 28/100