GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver

istio.io/service/server/response_latencies metric that HPA collected is different to Cloud monitoring

Open
#534 1 comment 0 reactions 0 assignees View on GitHub
custom-metrics-stackdriver-adapter
Dominant language
Go
Stars
409
Forks
236
Avg merge
2h 34m
Merged PRs (30d)
9

Description

Dear team,

I have an issue that the metric `istio.io/service/server/response_latencies` that HPA collected is different to the value I see on cloud monitoring. I have enabled Anthos service mesh on the cluster, and installed custom-metrics-stackdriver-adapter on the gke cluster. Already set `enable-distribution-support=true` for deployment custom-metrics-stackdriver-adapter to enable getting distribution metrics like istio.io/service/server/response_latencies. but the metric of `istio.io/service/server/response_latencies` that HPA collected is too high compares to that metric show on the cloud monitoring (for example: 6144s compares to 3.12s). Here is the information:
- GKE version: `v1.24.10-gke.2300`
- custom-metrics-stackdriver-adapter version: `gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.13.1-gke.0`
- custome-metric model: `adapter_new_resource_model.yaml`

HPA declaration:
```
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
annotations:
name: face-api-hpa
namespace: ekyc
spec:
maxReplicas: 1
metrics:
- external:
metric:
name: istio.io|service|server|response_latencies
selector:
matchLabels:
metric.labels.destination_service_name: face-api
reducer: REDUCE_PERCENTILE_50
target:
averageValue: "4"
type: AverageValue
type: External
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: face-api
```
The metrics that HPA collected as follow:
```
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
face-api-hpa Deployment/face-api 12697600m/4 (avg) 1 1 1 4d18h
```
The metric that show on monitoring:
![image](https://user-images.githubusercontent.com/19405290/232383316-1e6d1085-bba8-44a1-98bc-bfbbb85b44a5.png)

Is there anything wrong here or I mis-understand something?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.