Grafana-agent down in dashboard status
- Lenguaje dominante
- Go
- Estrellas
- 38
- Forks
- 9
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
#### Description
Grafana Agent shows down in dashboard status - newly deployed.
#### Screenshots


#### Log Messages
Grafana-agent logs (all periodically with similar error):
```
ts=2024-09-27T02:09:44.980753674Z level=error msg="non-recoverable error" component=prometheus.remote_write.default subcomponent=rw remote_name=bd527e url=http://prometheus.coder-observability.svc/api/v1/write count=2000 exemplarCount=0 err="server returned HTTP status 400 Bad Request: out of order sample"
ts=2024-09-27T02:09:50.086696001Z level=error msg="non-recoverable error" component=prometheus.remote_write.default subcomponent=rw remote_name=bd527e url=http://prometheus.coder-observability.svc/api/v1/write count=1558 exemplarCount=0 err="server returned HTTP status 400 Bad Request: out of order sample"
ts=2024-09-27T02:10:05.040293177Z level=error msg="non-recoverable error" component=prometheus.remote_write.default subcomponent=rw remote_name=bd527e url=http://prometheus.coder-observability.svc/api/v1/write count=626 exemplarCount=0 err="server returned HTTP status 400 Bad Request: out of order sample"
ts=2024-09-27T02:10:44.998642978Z level=error msg="non-recoverable error" component=prometheus.remote_write.default subcomponent=rw remote_name=bd527e url=http://prometheus.coder-observability.svc/api/v1/write count=2000 exemplarCount=0 err="server returned HTTP status 400 Bad Request: out of order sample"
ts=2024-09-27T02:11:05.079034516Z level=error msg="non-recoverable error" component=prometheus.remote_write.default subcomponent=rw remote_name=bd527e url=http://prometheus.coder-observability.svc/api/v1/write count=626 exemplarCount=0 err="server returned HTTP status 400 Bad Request: out of order sample"
```
Grafana pod logs:
```
logger=context userId=0 orgId=0 uname= t=2024-09-27T02:41:46.069582049Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.22.6 time_ms=0 duration=888.896µs size=105 referer= handler=/api/live/ws status_source=server
logger=context userId=0 orgId=0 uname= t=2024-09-27T02:42:07.000216216Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.22.6 time_ms=0 duration=759.171µs size=105 referer= handler=/api/live/ws status_source=server
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:13.869036352Z level=error msg="Cannot read directory" error="stat /tmp/dashboards: no such file or directory"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:13.869120292Z level=error msg="Failed to read content of symlinked path" path=/tmp/dashboards error="lstat /tmp/dashboards: no such file or directory"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:13.869133508Z level=info msg="falling back to original path due to EvalSymlink/Abs failure"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:13.869144257Z level=error msg="failed to search for dashboards" error="stat /tmp/dashboards: no such file or directory"
logger=context userId=0 orgId=0 uname= t=2024-09-27T02:42:27.003993831Z level=info msg="Request Completed" method=GET path=/api/live/ws status=401 remote_addr=10.0.22.6 time_ms=0 duration=667.764µs size=105 referer= handler=/api/live/ws status_source=server
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:43.868560334Z level=error msg="Cannot read directory" error="stat /tmp/dashboards: no such file or directory"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:43.868611097Z level=error msg="Failed to read content of symlinked path" path=/tmp/dashboards error="lstat /tmp/dashboards: no such file or directory"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:43.868620434Z level=info msg="falling back to original path due to EvalSymlink/Abs failure"
logger=provisioning.dashboard type=file name=sidecar t=2024-09-27T02:42:43.86862969Z level=error msg="failed to search for dashboards" error="stat /tmp/dashboards: no such file or directory"
```
#### To Reproduce
```
helm install coder-observability coder-observability/coder-observability \
--version 0.2.1 \
--namespace coder-observability \
--values values.yaml
```
* Note: Can share the values.yaml - latest and with updates only to postgres.
Guía de contribución
No hay ninguna guía de contribución indexada para este repositorio
Línea de trabajo
Empieza con el comando de instalación de Helm, el values.yaml proporcionado y los logs de los pods de Grafana Agent y Grafana. Rastrea los errores de remote-write de Prometheus y la ruta /tmp/dashboards que falta; después, confirma que el dashboard desplegado muestra el agente como up sin esos errores recurrentes.
Escrito por el modelo de indexación a partir del texto del issue.
Evaluación
- Stack tecnológico
- grafana, helm, kubernetes, prometheus
- Área
- observability
- Tipo de issue
- Error
- Dificultad
- 4/5
- Tiempo estimado
- 3-5 días
- Estado de actividad
- Estancado
- Claridad
- Necesita aclaración
- Aptitud para principiantes
- 35/100