GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver

Not able to use HPA with LoadBalancing cuso

Open
#424 1 comment 0 reactions 0 assignees View on GitHub
custom-metrics-stackdriver-adapter
Dominant language
Go
Stars
409
Forks
236
Avg merge
2h 34m
Merged PRs (30d)
9

Description

Hi,

I have installed the new adapter on GKE 1.20.6-gke.1000. I'm having an issue using HPA with loadbancing.googleapis.com/* metrics.

After installing the adapter i can see that the API Service is running and metrics are there

``` custom-metrics kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta1/' | jq '.resources[].name' | grep loadbalancing
"*/loadbalancing.googleapis.com|https|backend_request_bytes_count"
"*/loadbalancing.googleapis.com|https|backend_request_count"
"*/loadbalancing.googleapis.com|https|backend_response_bytes_count"
"*/loadbalancing.googleapis.com|https|internal|request_bytes_count"
"*/loadbalancing.googleapis.com|https|internal|request_count"
"*/loadbalancing.googleapis.com|https|internal|response_bytes_count"
"*/loadbalancing.googleapis.com|https|request_bytes_count"
"*/loadbalancing.googleapis.com|https|request_count"
"*/loadbalancing.googleapis.com|https|response_bytes_count"
"*/loadbalancing.googleapis.com|l3|external|egress_bytes_count"
"*/loadbalancing.googleapis.com|l3|external|egress_packets_count"
"*/loadbalancing.googleapis.com|l3|external|ingress_bytes_count"
"*/loadbalancing.googleapis.com|l3|external|ingress_packets_count"
"*/loadbalancing.googleapis.com|l3|internal|egress_bytes_count"
"*/loadbalancing.googleapis.com|l3|internal|egress_packets_count"
"*/loadbalancing.googleapis.com|l3|internal|ingress_bytes_count"
"*/loadbalancing.googleapis.com|l3|internal|ingress_packets_count"
"*/loadbalancing.googleapis.com|tcp_ssl_proxy|closed_connections"
"*/loadbalancing.googleapis.com|tcp_ssl_proxy|egress_bytes_count"
"*/loadbalancing.googleapis.com|tcp_ssl_proxy|ingress_bytes_count"
"*/loadbalancing.googleapis.com|tcp_ssl_proxy|new_connections"
"*/loadbalancing.googleapis.com|tcp_ssl_proxy|open_connections"```

But when i try to fetch a metric i get the following error

```
➜ custom-metrics kubectl get --raw '/apis/custom.metrics.k8s.io/v1beta2/namespaces/default/pods/*/loadbalancing.googleapis.com|https|request_count' | jq
Error from server: googleapi: Error 400: The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series., badRequest
```

I tried to check the logs of the custom-metrics-adapter in the custom-metrics namespace and i get the following error

```
E0614 15:49:51.162075 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0614 15:49:51.162623 1 writers.go:172] apiserver was unable to write a JSON response: http2: stream closed
E0614 15:49:51.163955 1 writers.go:172] apiserver was unable to write a JSON response: http2: stream closed
E0614 15:49:51.164639 1 writers.go:172] apiserver was unable to write a JSON response: http2: stream closed
E0614 15:49:51.164995 1 writers.go:172] apiserver was unable to write a JSON response: http2: stream closed
E0614 15:49:51.165324 1 writers.go:172] apiserver was unable to write a JSON response: http2: stream closed
E0614 15:49:51.165995 1 writers.go:172] apiserver was unable to write a JSON response: http2: stream closed
E0614 15:49:51.176632 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0614 15:49:51.253699 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0614 15:49:51.259983 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0614 15:49:51.261063 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0614 15:49:51.262300 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E0614 15:49:51.263484 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
```

Any idea what could be causing this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.