GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver

log spamming with horizontal pod autoscaler and custom-metrics-stackdriver-adapter

Open
#318 23 comments 23 reactions 0 assignees View on GitHub
custom-metrics-stackdriver-adapter
Dominant language
Go
Stars
409
Forks
236
Avg merge
2h 34m
Merged PRs (30d)
9

Description

I'm currently using `Horizontal Pod Autoscaler` (in google cloud) implemented with custom metrics, so `custom-metrics-stackdriver-adapter` is installed from [here](https://raw.githubusercontent.com/GoogleCloudPlatform/k8s-stackdriver/master/custom-metrics-stackdriver-adapter/deploy/production/adapter.yaml)

The problem is that it's generating more than 10 log messages a second with the following errors:

```json
jsonPayload: {
message: "apiserver was unable to write a JSON response: http2: stream closed"
pid: "1"
source: "writers.go:172"
}
```

and

```json
jsonPayload: {
message: "apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}"
pid: "1"
source: "status.go:71"
}
```

The HPA is working as expected, so the amount of errors is very strange and I couldn't found a reason for it, not could I find documentation on how to change this, or even change the amount of requests periodicity, not in HPA nor in this adapter.

HPA is configured as follows:

```yaml
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: xxx
namespace: xxx
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: xxx
minReplicas: 2
maxReplicas: 3
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 40
```

Kubernetes version is: 1.15

Is there any reason for this. It looks like a bug.

Also this issue seems to be related: https://github.com/GoogleCloudPlatform/k8s-stackdriver/issues/303

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.