GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver
log spamming with horizontal pod autoscaler and custom-metrics-stackdriver-adapter
- Dominant language
- Go
- Stars
- 409
- Forks
- 236
- Avg merge
- 2h 34m
- Merged PRs (30d)
- 9
Description
I'm currently using `Horizontal Pod Autoscaler` (in google cloud) implemented with custom metrics, so `custom-metrics-stackdriver-adapter` is installed from [here](https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml)
The problem is that it's generating more than 10 log messages a second with the following errors:
```json
jsonPayload: {
message: "apiserver was unable to write a JSON response: http2: stream closed"
pid: "1"
source: "writers.go:172"
}
```
and
```json
jsonPayload: {
message: "apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}"
pid: "1"
source: "status.go:71"
}
```
The HPA is working as expected, so the amount of errors is very strange and I couldn't found a reason for it, not could I find documentation on how to change this, or even change the amount of requests periodicity, not in HPA nor in this adapter.
HPA is configured as follows:
```yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: xxx
namespace: xxx
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: xxx
minReplicas: 2
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 40
```
Kubernetes version is: 1.15
Is there any reason for this. It looks like a bug.
Also this issue seems to be related: https://github.com/GoogleCloudPlatform/k8s-stackdriver/issues/303
Contributor guide
Assessment
This issue has not been assessed yet.