canonical / canonical/observability-stack

Add troubleshooting guide for timeseries with NaN

Open
#164 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
HCL
Stars
9
Forks
12
Avg merge
2d 40m
Merged PRs (30d)
3

Description

Via @jameinel via [matrix](https://matrix.to/#/!nHXpRkcSNJHlHdUGbQ:ubuntu.com/$M3ZLzis0qY0p56nWg8AgV-8T_W9WgZIBzlACtm2wFX8?via=ubuntu.com&via=matrix.org&via=mozilla.org).

When timeseries have `NaN`s,

```
avg_over_time(
juju_apiserver_request_duration_seconds{
juju_controller="$controller",
juju_unit=~"$controller_host",
method="FullStatus",
quantile="0.99",error_code!~"not found|unauthorized access"
}[5m]
)
```
Image

then aggregation operations produce unexpected results:

```
avg without (version, error_code)
(
juju_apiserver_request_duration_seconds{
juju_controller="$controller",
juju_unit=~"$controller_host",
method="FullStatus",
quantile="0.99",error_code!~"not found|unauthorized access"
}[5m]
)
```
Image

and only when we filter out the `NaN`s with e.g.
```
avg(avg_over_time((juju_apiserver_request_duration_seconds > 0.0001)[1h:]))
```

then we get the expected results.

We should document:
- How to avoid `NaN`s at instrumentation time.
- Treating potential `NaN` in dashboard/alert `expr`s.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.