aws / aws/aws-xray-sdk-python

Trace Size Exceeded

Abierto
#309 4 comentarios 0 reacciones 0 asignados Ver en GitHub
question stale
Lenguaje dominante
Python
Estrellas
338
Forks
147
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

Hi,

I am starting to see this error a lot in most of our django applications.

```
[Error] Unprocessed segment: {
ErrorCode: "InvalidSegment",
Id: "xxxxxxxxxxxxxxxxxx",
Message: "Trace Size Exceeded. Reduce the number of segments per trace and the size of your segment documents."
}
```

Also, whilst looking at the logs:
```
2021-07-21T11:37:15.068+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.016 seconds)
2021-07-21T11:37:15.071+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.087+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.102+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.117+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.135+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.007 seconds)
2021-07-21T11:37:15.148+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.163+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.178+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.194+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.214+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.225+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.239+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.256+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.288+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.017 seconds)
2021-07-21T11:37:15.289+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.311+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.326+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.336+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.350+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.365+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.379+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.395+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.415+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.010 seconds)
2021-07-21T11:37:15.424+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.439+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.458+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.009 seconds)
2021-07-21T11:37:15.467+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.483+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.497+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.513+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.534+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.554+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.011 seconds)
2021-07-21T11:37:15.563+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.577+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.593+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.005 seconds)
2021-07-21T11:37:15.610+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.007 seconds)
2021-07-21T11:37:15.624+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.004 seconds)
2021-07-21T11:37:15.641+03:00 2021-07-21T08:37:15Z [Info] Successfully sent batch of 50 segments (0.006 seconds)
```
It looks like a lot of segments are being sent while nothing is really happening on the django application itself.

We are running the daemon on the alongside the django application, in the same docker container. We have the recorder setup as instructed [here](https://docs.aws.amazon.com/xray/latest/devguide/xray-sdk-python-configuration.html#xray-sdk-python-middleware-configuration-django):
```
XRAY_RECORDER = {
"AUTO_INSTRUMENT": True,
"AWS_XRAY_CONTEXT_MISSING": "LOG_ERROR",
"AWS_XRAY_TRACING_NAME": f"{env('PARTNER', default='embed')}-{env('ENVIRONMENT_TYPE')}",
"PLUGINS": ("EC2Plugin", "ECSPlugin"),
"AWS_XRAY_SDK_ENABLED": env("AWS_XRAY_SDK_ENABLED", default=False),
}
```
I am also seeing this when the application starts:
```
[INFO] No effective centralized sampling rule match. Fallback to local rules.
```
Our sampling rules are not local tho, but defined in the xray we console.

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Comience con la configuración de Django recorder y middleware mostrada en el issue y con la documentación de configuración de AWS X-Ray Python a la que enlaza. Revise las entradas de registro repetidas de 50 segmentos del daemon junto con el mensaje de fallback del muestreo centralizado; el trabajo estará hecho cuando se haya identificado la causa del error de tamaño del trace y se haya documentado una resolución aplicable.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
aws, django, docker, python
Área
observability-sre
Tipo de issue
Error
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Estancado
Claridad
Necesita aclaración
Aptitud para principiantes
28/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.