GoogleCloudPlatform / GoogleCloudPlatform/k8s-stackdriver

Metrics not sent after certain amount of time using prometheus-to-sd

Open
#202 5 comments 21 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
409
Forks
236
Avg merge
2h 34m
Merged PRs (30d)
9

Description

I am using prometheus-to-sd v0.3.1 for sending metrics from a spring boot application.
When the application is running in isolation i.e. no service hitting this service, it is able to send metrics to Stackdriver. But as soon as i switch the traffic to this service i.e. other service starts calling this service it stops sending metrics.
Also if I let the application to run in isolation for few hours it stops sending metrics after 3-4 hours.

When i check the logs in Stackdriver logging interface there is no logs form prometheus-to-sd container from the point at which its stop sending metrics.
Could let me know where i could find more info for this problem?

Gke cluster version: 1.10.6.2
kubernetes version: 1.10


```
- name: applicationName-sd
image: gcr.io/google-containers/prometheus-to-sd:v0.3.1
command:
- /monitor
- --source=:http://localhost:42802/prometheus
- --stackdriver-prefix=custom.googleapis.com/applicationName
- --pod-id=$(POD_ID)
- --namespace-id=$(POD_NAMESPACE)
env:
- name: POD_ID
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
```

applicationName and port is having right names in my config.

Logs from application start up
```
`GCE config: &{Project:projectName Zone:zone Cluster:dev-v2 Instance:insatnceName MetricsPrefix:custom.googleapis.com/applicationName}
Taking source configs from flags
Taking source configs from kubernetes api server
Built the following source configs: [{ localhost 42802 /prometheus [] {applciationName-5f75979b6b-b4jh6 namespace}}]
Running prometheus-to-sd, monitored target is localhost:42802
GCE config: &{Project:projectName Zone:us-east4-b Cluster:dev-v2 Instance:gke-dev-v2-standard8-10c2b7d3-xx6m MetricsPrefix:custom.googleapis.com/applicationName}
Taking source configs from flags
Taking source configs from kubernetes api server
Built the following source configs: [{ localhost 42802 /prometheus [] {applicationName-5f75979b6b-n5tt6 namespace}}]
Running prometheus-to-sd, monitored target is localhost:42802
GCE config: &{Project:projectName Zone:us-east4-b Cluster:dev-v2 Instance:gke-dev-v2-standard8-10c2b7d3-d6dd MetricsPrefix:custom.googleapis.com/applicationName}
Taking source configs from flags
Taking source configs from kubernetes api server
Built the following source configs: [{ localhost 42802 /prometheus [] {applicationName-5f75979b6b-5fdlb namespace}}]
Running prometheus-to-sd, monitored target is localhost:42802
GCE config: &{Project:projectName Zone:us-east4-c Cluster:dev-v2 Instance:gke-dev-v2-standard8-541c8362-kv5l MetricsPrefix:custom.googleapis.com/applicationName}
Taking source configs from flags
Taking source configs from kubernetes api server
Built the following source configs: [{ localhost 42802 /prometheus [] {applicationName-5f75979b6b-tqjcw namespace}}]
Running prometheus-to-sd, monitored target is localhost:42802
GCE config: &{Project:projectName Zone:us-east4-a Cluster:dev-v2 Instance:gke-dev-v2-standard8-61da8de0-zb18 MetricsPrefix:custom.googleapis.com/applicationName}
Taking source configs from flags
Taking source configs from kubernetes api server
Built the following source configs: [{ localhost 42802 /prometheus [] {applicationName-5f75979b6b-74vnt namespace}}]
Running prometheus-to-sd, monitored target is localhost:42802
GCE config: &{Project:projectName Zone:us-east4-b Cluster:dev-v2 Instance:gke-dev-v2-standard8-10c2b7d3-5scm MetricsPrefix:custom.googleapis.com/applicationName}
Taking source configs from flags
Taking source configs from kubernetes api server
Built the following source configs: [{ localhost 42802 /prometheus [] {applicationName-5f75979b6b-l46x8 namespace}}]
Running prometheus-to-sd, monitored target is localhost:42802
Metric process_start_time_seconds invalid or not defined. Using 1970-01-01 00:00:01 +0000 UTC instead. Cumulative metrics might be inaccurate.`
```

There are warning messages like these after start up but it was still sending the metrics to stackdriver.
`Metric process_start_time_seconds invalid or not defined. Using 1970-01-01 00:00:01 +0000 UTC instead. Cumulative metrics might be inaccurate`

The port that is exposing the metrics is also the port for liveliness and readiness probe but it hits a different url.

I have the 3+ pods running.
Let me what more details i could provide.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.