GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver
istio.io/service/server/response_latencies metric that HPA collected is different to Cloud monitoring
- Dominant language
- Go
- Stars
- 409
- Forks
- 236
- Avg merge
- 2h 34m
- Merged PRs (30d)
- 9
Description
Dear team,
I have an issue that the metric `istio.io/service/server/response_latencies` that HPA collected is different to the value I see on cloud monitoring. I have enabled Anthos service mesh on the cluster, and installed custom-metrics-stackdriver-adapter on the gke cluster. Already set `enable-distribution-support=true` for deployment custom-metrics-stackdriver-adapter to enable getting distribution metrics like istio.io/service/server/response_latencies. but the metric of `istio.io/service/server/response_latencies` that HPA collected is too high compares to that metric show on the cloud monitoring (for example: 6144s compares to 3.12s). Here is the information:
- GKE version: `v1.24.10-gke.2300`
- custom-metrics-stackdriver-adapter version: `gcr.io/gke-release/custom-metrics-stackdriver-adapter:v0.13.1-gke.0`
- custome-metric model: `adapter_new_resource_model.yaml`
HPA declaration:
```
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
annotations:
name: face-api-hpa
namespace: ekyc
spec:
maxReplicas: 1
metrics:
- external:
metric:
name: istio.io|service|server|response_latencies
selector:
matchLabels:
metric.labels.destination_service_name: face-api
reducer: REDUCE_PERCENTILE_50
target:
averageValue: "4"
type: AverageValue
type: External
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: face-api
```
The metrics that HPA collected as follow:
```
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
face-api-hpa Deployment/face-api 12697600m/4 (avg) 1 1 1 4d18h
```
The metric that show on monitoring:

Is there anything wrong here or I mis-understand something?
Contributor guide
Assessment
This issue has not been assessed yet.