GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver
[prometheus-to-sd] Error while sending request to Stackdriver googleapi: Error 400: One or more TimeSeries could not be written
- Dominant language
- Go
- Stars
- 409
- Forks
- 236
- Avg merge
- 2h 34m
- Merged PRs (30d)
- 9
Description
Getting this error in the logs every minute. Starts appearing a few minutes after the cluster creation.
Here's the full output:
```
Error while sending request to Stackdriver googleapi: Error 400: One or more TimeSeries could not be written: Unknown metric: kubernetes.io/internal/addons/workload_identity/go_gc_duration_seconds_count: timeSeries[31]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_gc_duration_seconds_sum: timeSeries[40]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_goroutines: timeSeries[33]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_info: timeSeries[5]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_alloc_bytes: timeSeries[7]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_alloc_bytes_total: timeSeries[18]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_buck_hash_sys_bytes: timeSeries[53]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_frees_total: timeSeries[46]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_gc_cpu_fraction: timeSeries[10]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_gc_sys_bytes: timeSeries[51]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_alloc_bytes: timeSeries[50]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_idle_bytes: timeSeries[4]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_inuse_bytes: timeSeries[54]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_objects: timeSeries[43]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_released_bytes: timeSeries[49]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_heap_sys_bytes: timeSeries[6]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_last_gc_time_seconds: timeSeries[55]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_lookups_total: timeSeries[56]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_mallocs_total: timeSeries[20]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_mcache_inuse_bytes: timeSeries[52]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_mcache_sys_bytes: timeSeries[19]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_mspan_inuse_bytes: timeSeries[44]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_mspan_sys_bytes: timeSeries[17]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_next_gc_bytes: timeSeries[48]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_other_sys_bytes: timeSeries[35]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_stack_inuse_bytes: timeSeries[39]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_stack_sys_bytes: timeSeries[45]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_memstats_sys_bytes: timeSeries[34]; Unknown metric: kubernetes.io/internal/addons/workload_identity/go_threads: timeSeries[41]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_cpu_seconds_total: timeSeries[1]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_max_fds: timeSeries[32]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_open_fds: timeSeries[47]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_resident_memory_bytes: timeSeries[42]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_start_time_seconds: timeSeries[3]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_virtual_memory_bytes: timeSeries[30]; Unknown metric: kubernetes.io/internal/addons/workload_identity/process_virtual_memory_max_bytes: timeSeries[38]; Unknown metric: kubernetes.io/internal/addons/workload_identity/promhttp_metric_handler_requests_in_flight: timeSeries[2]; Unknown metric: kubernetes.io/internal/addons/workload_identity/promhttp_metric_handler_requests_total: timeSeries[27-29]; Value type for metric kubernetes.io/internal/addons/workload_identity/metadata_server_build_info must be DOUBLE, but is INT64.: timeSeries[0], badRequest
```
`source: "stackdriver.go:60" `
This is the GKE cluster config I'm using:
```
{
"cluster": {
"name": "test-cluster",
"masterAuth": {
"clientCertificateConfig": {}
},
"network": "projects/REDACTED/global/networks/default",
"addonsConfig": {
"httpLoadBalancing": {
"disabled": true
},
"horizontalPodAutoscaling": {},
"kubernetesDashboard": {
"disabled": true
},
"networkPolicyConfig": {},
"istioConfig": {
"disabled": true
}
},
"subnetwork": "projects/REDACTED/regions/europe-west4/subnetworks/default",
"nodePools": [
{
"name": "default-pool",
"config": {
"machineType": "g1-small",
"diskSizeGb": 10,
"oauthScopes": [
"https://www.googleapis.com/auth/devstorage.read_only",
"https://www.googleapis.com/auth/logging.write",
"https://www.googleapis.com/auth/monitoring",
"https://www.googleapis.com/auth/servicecontrol",
"https://www.googleapis.com/auth/service.management.readonly",
"https://www.googleapis.com/auth/trace.append"
],
"metadata": {
"disable-legacy-endpoints": "true"
},
"imageType": "COS",
"diskType": "pd-standard",
"shieldedInstanceConfig": {
"enableSecureBoot": true,
"enableIntegrityMonitoring": true
}
},
"initialNodeCount": 2,
"autoscaling": {},
"management": {
"autoUpgrade": true,
"autoRepair": true
},
"version": "1.15.8-gke.2",
"upgradeSettings": {
"maxSurge": 1
}
}
],
"networkPolicy": {
"provider": "CALICO",
"enabled": true
},
"ipAllocationPolicy": {
"useIpAliases": true
},
"masterAuthorizedNetworksConfig": {},
"defaultMaxPodsConstraint": {
"maxPodsPerNode": "110"
},
"authenticatorGroupsConfig": {},
"privateClusterConfig": {},
"databaseEncryption": {
"keyName": "projects/REDACTED/locations/europe-west4/keyRings/REDACTED",
"state": "ENCRYPTED"
},
"shieldedNodes": {
"enabled": true
},
"workloadIdentityConfig": {
"identityNamespace": "REDACTED"
},
"clusterTelemetry": {
"type": "ENABLED"
},
"initialClusterVersion": "1.15.8-gke.2",
"location": "europe-west4-a"
}
}
```
Contributor guide
Assessment
This issue has not been assessed yet.