prometheus-community / prometheus-community/postgres_exporter

autoDiscoverDatabases:true gives error 500 (collected metric "pg_stat_activity_idle_process_seconds" was collected before)

Open
#753 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
3.6k
Forks
835
Avg merge
2d 4h
Merged PRs (30d)
10

Description

Copied from prometheus-community/helm-charts

This issue originally was opened here but was unattended. Copied here as is. I experience this issue with postgres_exporter 0.10.1 and latest 0.11.1 as well

Describe the bug a clear and concise description of what the bug is.

With config.autoDiscoverDatabases: true I got error 500 on /metrics endpoint:

< HTTP/1.1 500 Internal Server Error
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Thu, 14 Jul 2022 11:37:11 GMT
< Transfer-Encoding: chunked
< 
An error has occurred while serving metrics:

12 error(s) occurred:
* collected metric "pg_stat_activity_idle_process_seconds" { label:<name:"application_name" value:"" > label:<name:"server" value:"develop-postgresql:5432" > histogram:<sample_count:117 sample_sum:9240 bucket:<cumulative_count:36 upper_bound:1 > bucket:<cumulative_count:42 upper_bound:2 > bucket:<cumulative_count:53 upper_bound:5 > bucket:<cumulative_count:64 upper_bound:15 > bucket:<cumulative_count:65 upper_bound:30 > bucket:<cumulative_count:67 upper_bound:60 > bucket:<cumulative_count:67 upper_bound:90 > bucket:<cumulative_count:79 upper_bound:120 > bucket:<cumulative_count:118 upper_bound:300 > > } was collected before with the same name and label values
* collected metric "pg_stat_activity_idle_process_seconds" { label:<name:"application_name" value:"Patroni" > label:<name:"server" value:"develop-postgresql:5432" > histogram:<sample_count:1 sample_sum:1 bucket:<cumulative_count:0 upper_bound:1 > bucket:<cumulative_count:1 upper_bound:2 > bucket:<cumulative_count:1 upper_bound:5 > bucket:<cumulative_count:1 upper_bound:15 > bucket:<cumulative_count:1 upper_bound:30 > bucket:<cumulative_count:1 upper_bound:60 > bucket:<cumulative_count:1 upper_bound:90 > bucket:<cumulative_count:1 upper_bound:120 > bucket:<cumulative_count:1 upper_bound:300 > > } was collected before with the same name and label values

According to https://github.com/prometheus-community/postgres_exporter/issues/429#issuecomment-850865088 I manually added key master: true to query pg_stat_activity_idle and it fixes error

What's your helm version?

version.BuildInfo{Version:"v3.7.1", GitCommit:"1d11fcb5d3f3bf00dbe6fe31b8412839a96b3dc4", GitTreeState:"clean", GoVersion:"go1.16.9"}

What's your kubectl version?

Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:10:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}

Which chart?

prometheus-postgres-exporter

What's the chart version?

3.0.3

What happened?

No response

What you expected to happen?

No response

How to reproduce it?

No response

Enter the changed values of values.yaml?

serviceMonitor:
enabled: true
config:
datasource:
host: develop-postgresql
user: develop_admin
password:
passwordSecret:
name: develop-admin.develop-postgresql.credentials.postgresql.acid.zalan.do
key: password
database: 'template1'
logLevel: info
autoDiscoverDatabases: true

Enter the command that you execute and failing/misfunctioning.

helm install postgres-exporter prometheus-community/prometheus-postgres-exporter --namespace db -f ./postrges_exporter.yaml

Anything else we need to know?

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the /metrics endpoint and the pg_stat_activity_idle query involved in autoDiscoverDatabases. Inspect how that query is handled across discovered databases and compare it with the reported manual master: true change. Done means the exporter serves metrics without duplicate pg_stat_activity_idle_process_seconds errors when autoDiscoverDatabases is enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, helm, kubernetes, postgresql, prometheus
Domain
databases, devops, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.