GoogleCloudPlatform / GoogleCloudPlatform/prometheus-engine

Improve error handling when translating classic histograms to distribution (e.g. parsing bucket "le" label edge cases)

Open
#982 3 comments 0 reactions 0 assignees View on GitHub
bug help wanted
Dominant language
Go
Stars
232
Forks
109
Avg merge
1d 3h
Merged PRs (30d)
27

Description

We got one Cx getting error from the CMP: `One or more TimeSeries could not be written: Field timeSeries[0].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| entry 1 has a value of 0.005 which is less than the value of entry 0 which is 0.005`. This indicates that either:

A. Client exposes classic histogram series with two series for the same `le` value. This is impossible as duplicated series will be handled by scraping logic.
B. Our [conversion from string le labels to float](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/5f75ba3d5ae6477169b54543b3266e4d3fe22d7b/pkg/export/transform.go#L444) buckets ended up as the same `0.005` buckets. This can totally happen on extreme bucket `le` values like `"0.00500000000000000006"` and `"0.005"` in subsequent classic series. See [repro](https://go.dev/play/p/SG5SGQkKwhD). Furthermore [check does not validate for "0" diffs, only negative diffs on it](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/5f75ba3d5ae6477169b54543b3266e4d3fe22d7b/pkg/export/transform.go#L317).
C. Rounding error no CMP side. Unlikely as CMP accepts double value as well.

### Action Items

* https://github.com/GoogleCloudPlatform/prometheus-engine/blob/5f75ba3d5ae6477169b54543b3266e4d3fe22d7b/pkg/export/transform.go#L317 diff betwen buckets can't be zero too, not only negative.
* We catch that problem when we parse to float to see if it's scraping/client issue or parsing to float precision issue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.