influxdata / influxdata/influxdb
Chronograf Not Rendering the Query Result
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
We are running a POC to migrate VM running influx version 1 to new and shinny influx version 2.0 on kubernetes. For that purpose, we are running default config of influx on kubernetes which is ingesting data pushed by telegraf after reading from a kafka topic. Ingestion is absolutely fine, but the issue arises when we try to view the metrics on the chronograf gui. It errors out by either freezing the tab or by killing the influxdb pod due to unresponsiveness.
Interestingly; if we run the exact same query via Grafana, it shows the graph immediately; which indicates that the issue is not with the influx but the chronograf itself.
Please see below the details.
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Run influxdb2 on kubernetes using helm charts (4G Memory and 4 CPU)
2. Run telegarf on kubernetes same namespace using helm chart to read from kafka and push into influx
3. Run influx-stress on a VM to push metrics from tool to influx via telegraf --> kafka --> telegraf --> influx
4. Run following simple query to explore the data from influx-stress tool
```sh
from(bucket: "kafka_data")
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "ctr")
|> filter(fn: (r) => r["_field"] == "n")
|> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
|> yield(name: "mean")
```
__Expected behavior:__
A line graph to show the metrics being pushed from influx-stress tool
__Actual behavior:__
There are 2 behaviors that were noticed
When running without ISTIO
1. The Chronograph gui just gets stuck while rendering and eventually browser tries to kill the tab due to unresponsiveness.
When running with ISTIO
2. Influxdb pod consumes all available memory to it and eventually the health and readiness probes fails due to which pod gets restarted
```sh
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Wed, 23 Dec 2020 17:08:33 +0000
Finished: Tue, 29 Dec 2020 11:32:58 +0000
```
Events
```sh
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning Unhealthy 79s (x7 over 5d18h) kubelet, hostname.mycompany.local Readiness probe failed: Get http://192.168.220.161:15021/healthz/ready: net/http: req
uest canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 78s (x4 over 5d18h) kubelet, hostname.mycompany.local Liveness probe failed: Get http://192.168.220.161:15020/app-health/influxdb2/livez: n
et/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 76s (x2 over 5d18h) kubelet, hostname.mycompany.local Readiness probe failed: Get http://192.168.220.161:15020/app-health/influxdb2/readyz:
net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 50s (x4 over 5d18h) kubelet, hostname.mycompany.local Readiness probe failed: Get http://192.168.220.161:15021/healthz/ready: net/http: req
uest canceled (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 49s (x6 over 5d18h) kubelet, hostname.mycompany.local Readiness probe failed: Get http://192.168.220.161:15020/app-health/influxdb2/readyz:
net/http: request canceled (Client.Timeout exceeded while awaiting headers)
```
__Environment info:__
* System info: On Pre kubernetes
* InfluxDB version: influxdb2 OSS version deployed using helm chart 1.0.13
* Other relevant environment details: Container runtime --> docker
__Config:__
No custom config; we are using default config, only added config is influx token, bucket etc
__Logs:__
Include snippet of errors in log.
```sh
ts=2021-01-04T11:40:25.435747Z lvl=info msg="Error writing response to client" log_id=0RVB0BSl000 handler=flux error="csv encoder error: write tcp 192.168.212.78:808
6->192.168.159.198:46776: write: broken pipe"
ts=2021-01-04T11:40:59.623752Z lvl=info msg="Error writing response to client" log_id=0RVB0BSl000 handler=flux error="csv encoder error: write tcp 192.168.212.78:808
6->192.168.159.198:46988: write: broken pipe"
ts=2021-01-04T11:43:56.420716Z lvl=info msg="Error writing response to client" log_id=0RVB0BSl000 handler=flux error="csv encoder error: write tcp 192.168.212.78:808
6->192.168.159.198:48084: write: broken pipe"
ts=2021-01-04T11:45:16.464803Z lvl=info msg="Error writing response to client" log_id=0RVB0BSl000 handler=flux error="csv encoder error: write tcp 192.168.212.78:808
6->192.168.159.198:48570: write: broken pipe"
ts=2021-01-04T11:47:06.535610Z lvl=info msg="Error writing response to client" log_id=0RVB0BSl000 handler=flux error="csv encoder error: write tcp 192.168.212.78:808
6->192.168.159.198:49244: write: broken pipe"
ts=2021-01-04T11:48:24.428184Z lvl=info msg="Error writing response to client" log_id=0RVB0BSl000 handler=flux error="csv encoder error: write tcp 192.168.212.78:808
6->192.168.159.198:49720: write: broken pipe"
```
__Performance:__
Generate profiles with the following commands for bugs related to performance, locking, out of memory (OOM), etc.
Contributor guide
Research direction
Start by reproducing the Flux query in Chronograf with the stated Kubernetes, Helm, Telegraf, Kafka, and InfluxDB setup, then compare it with Grafana. Review the provided InfluxDB logs and pod events, including the broken-pipe errors and exit code 137. Done means the query renders a line graph without freezing the browser or exhausting the InfluxDB pod.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, grafana, helm, kafka, kubernetes
- Domain
- databases, infrastructure, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100