canonical / canonical/observability-stack
Add troubleshooting guide for timeseries with NaN
- Dominant language
- HCL
- Stars
- 9
- Forks
- 12
- Avg merge
- 2d 40m
- Merged PRs (30d)
- 3
Description
Via @jameinel via [matrix](https://matrix.to/#/!nHXpRkcSNJHlHdUGbQ:ubuntu.com/$M3ZLzis0qY0p56nWg8AgV-8T_W9WgZIBzlACtm2wFX8?via=ubuntu.com&via=matrix.org&via=mozilla.org).
When timeseries have `NaN`s,
```
avg_over_time(
juju_apiserver_request_duration_seconds{
juju_controller="$controller",
juju_unit=~"$controller_host",
method="FullStatus",
quantile="0.99",error_code!~"not found|unauthorized access"
}[5m]
)
```
then aggregation operations produce unexpected results:
```
avg without (version, error_code)
(
juju_apiserver_request_duration_seconds{
juju_controller="$controller",
juju_unit=~"$controller_host",
method="FullStatus",
quantile="0.99",error_code!~"not found|unauthorized access"
}[5m]
)
```
and only when we filter out the `NaN`s with e.g.
```
avg(avg_over_time((juju_apiserver_request_duration_seconds > 0.0001)[1h:]))
```
then we get the expected results.
We should document:
- How to avoid `NaN`s at instrumentation time.
- Treating potential `NaN` in dashboard/alert `expr`s.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.