google / google/slo-generator

🐛 [BUG] - incorrect calculation for prometheus provider

Open
#319 7 comments 0 reactions 1 assignee Claimed by @lvaylet View on GitHub
bug triage
Dominant language
Python
Stars
564
Forks
88
PR merge metrics
No merged PRs in 30d

Description

### SLO Generator Version

v2.3.3

### Python Version

3.9.13

### What happened?

I am using `ServiceLevelObjective` in `sre.google.com/v2` with this spec
```yaml
spec:
backend: prometheus
method: good_bad_ratio
service_level_indicator:
filter_good: envoy_cluster_external_upstream_rq{app="x",envoy_cluster_name="y",kubernetes_namespace="z",envoy_response_code!~"5.."}
filter_valid: envoy_cluster_external_upstream_rq{app="x",envoy_cluster_name="y",kubernetes_namespace="z"}
```

calculation ends with `SLI is not between 0 and 1 (value = 1.000091)`

with `DEBUG=1` it seems that for 100% SLI prometheus returns sometime `filter_good > filter_valid` the DEBUG logs
```
....
slo_generator.backends.prometheus - DEBUG - Query: sum(increase(envoy_cluster_external_upstream_rq{app="x",envoy_cluster_name="y",kubernetes_namespace="z",envoy_response_code!~"5.."}[3600s]))
slo_generator.backends.prometheus - DEBUG - {'data': {'result': [{'metric': {},
'value': [1674577674.326, '44.197872287004095']}],
'resultType': 'vector'},
'status': 'success'}
...
slo_generator.backends.prometheus - DEBUG - Query: sum(increase(envoy_cluster_external_upstream_rq{app="x",envoy_cluster_name="y",kubernetes_namespace="z"}[3600s]))
slo_generator.backends.prometheus - DEBUG - {'data': {'result': [{'metric': {},
'value': [1674577674.959, '44.193849641808484']}],
'resultType': 'vector'},
'status': 'success'}
slo_generator.backends.prometheus - DEBUG - Good events: 44.197872287004095 | Bad events: -0.004022645195611574
slo_generator.report - DEBUG - paket-static-availability | 1 hour | Backend response: (44.197872287004095, -0.004022645195611574)
```

### What did you expect?

calculation of SLI return 1

### Code of Conduct

- [X] I agree to follow this project's Code of Conduct

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.