google / google/slo-generator

Add 'offset' field to handle metrics ingestion delays

Open
#111 11 comments 0 reactions 1 assignee Claimed by @lvaylet View on GitHub
bug help wanted releases/v3
Dominant language
Python
Stars
564
Forks
88
PR merge metrics
No merged PRs in 30d

Description

Hi,

I'm currently trying to understand why the SLO-pipeline sees "bad events" (backend "Stackdriver") although there were no bad events when looking at the actual underlying Stackdriver metric (see screenshot below).

Example: Requests with code `201` are defined as good events and valid events are `201` + `500..511`. The chart with the log-based metrics shows that there are only requests with code `2xx` and `4xx`, but still the generator often exports metrics with "bad events" > 0 (also values for burn rate and SLI are effected).

Note: The service has a higher rate of incoming requests which should provide a meaningful signal during the alerting windows

Any idea what can cause this?

SLO configuration for feature "carts-create"
```
service_name: projects
feature_name: carts-create
slo_description: 95% of cart create API HTTP responses are successful
slo_name: availability
slo_target: 0.95
metadata: {}
backend:
class: Stackdriver
method: good_bad_ratio
project_id: ${stackdriver_host_project_id}
measurement:
filter_good: >
project=${project_id}
metric.type="logging.googleapis.com/user/carts_create_requests_total"
metric.labels.http_status = 201
filter_valid: >
project=${project_id}
metric.type="logging.googleapis.com/user/carts_create_requests_total"
( metric.labels.http_status = 201 OR
metric.labels.http_status = 500 OR
metric.labels.http_status = 501 OR
metric.labels.http_status = 502 OR
metric.labels.http_status = 503 OR
metric.labels.http_status = 504 OR
metric.labels.http_status = 505 OR
metric.labels.http_status = 506 OR
metric.labels.http_status = 507 OR
metric.labels.http_status = 508 OR
metric.labels.http_status = 509 OR
metric.labels.http_status = 510 OR
metric.labels.http_status = 511 )
exporters:
- class: Stackdriver
project_id: ${stackdriver_host_project_id}
```

Metrics
- left: error budget rate (feature: "carts-create")
- right: request count split by label `http_status` -> no 5xx,
Bildschirmfoto 2020-11-25 um 15 36 59

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.