GoogleCloudPlatform / GoogleCloudPlatform/prometheus-engine
Improve error handling when translating classic histograms to distribution (e.g. parsing bucket "le" label edge cases)
- Dominant language
- Go
- Stars
- 232
- Forks
- 109
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 27
Description
We got one Cx getting error from the CMP: `One or more TimeSeries could not be written: Field timeSeries[0].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| entry 1 has a value of 0.005 which is less than the value of entry 0 which is 0.005`. This indicates that either:
A. Client exposes classic histogram series with two series for the same `le` value. This is impossible as duplicated series will be handled by scraping logic.
B. Our [conversion from string le labels to float](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/5f75ba3d5ae6477169b54543b3266e4d3fe22d7b/pkg/export/transform.go#L444) buckets ended up as the same `0.005` buckets. This can totally happen on extreme bucket `le` values like `"0.00500000000000000006"` and `"0.005"` in subsequent classic series. See [repro](https://go.dev/play/p/SG5SGQkKwhD). Furthermore [check does not validate for "0" diffs, only negative diffs on it](https://github.com/GoogleCloudPlatform/prometheus-engine/blob/5f75ba3d5ae6477169b54543b3266e4d3fe22d7b/pkg/export/transform.go#L317).
C. Rounding error no CMP side. Unlikely as CMP accepts double value as well.
### Action Items
* https://github.com/GoogleCloudPlatform/prometheus-engine/blob/5f75ba3d5ae6477169b54543b3266e4d3fe22d7b/pkg/export/transform.go#L317 diff betwen buckets can't be zero too, not only negative.
* We catch that problem when we parse to float to see if it's scraping/client issue or parsing to float precision issue.
Contributor guide
Assessment
This issue has not been assessed yet.